


FLEX(1)                                                   FLEX(1)


NNAAMMEE
       flex - fast lexical analyzer generator

SSYYNNOOPPSSIISS
       fflleexx [[--bbccddffiinnppssttvvFFIILLTT88 --CC[[eeffmmFF]] --SSsskkeelleettoonn]] _[_f_i_l_e_n_a_m_e _._._._]

DDEESSCCRRIIPPTTIIOONN
       _f_l_e_x is a tool for  generating  _s_c_a_n_n_e_r_s_:  programs  which
       recognized lexical patterns in text.  _f_l_e_x reads the given
       input files, or its standard input if no  file  names  are
       given,  for  a  description of a scanner to generate.  The
       description is in the form of pairs of regular expressions
       and  C  code,  called  _r_u_l_e_s_. _f_l_e_x generates as output a C
       source file, lleexx..yyyy..cc,, which defines  a  routine  yyyylleexx(())..
       This  file is compiled and linked with the --llffll library to
       produce an executable.  When the  executable  is  run,  it
       analyzes  its input for occurrences of the regular expres-
       sions.  Whenever it finds one, it executes the correspond-
       ing C code.

SSOOMMEE SSIIMMPPLLEE EEXXAAMMPPLLEESS
       First  some  simple  examples to get the flavor of how one
       uses _f_l_e_x_.  The following _f_l_e_x input specifies  a  scanner
       which  whenever  it  encounters the string "username" will
       replace it with the user's login name:

           %%
           username    printf( "%s", getlogin() );

       By default, any text not matched  by  a  _f_l_e_x  scanner  is
       copied to the output, so the net effect of this scanner is
       to copy its input file to its output with each  occurrence
       of  "username" expanded.  In this input, there is just one
       rule.  "username" is the _p_a_t_t_e_r_n and the "printf"  is  the
       _a_c_t_i_o_n_.  The "%%" marks the beginning of the rules.

       Here's another simple example:

               int num_lines = 0, num_chars = 0;

           %%
           \n    ++num_lines; ++num_chars;
           .     ++num_chars;

           %%
           main()
               {
               yylex();
               printf( "# of lines = %d, # of chars = %d\n",
                       num_lines, num_chars );
               }

       This  scanner counts the number of characters and the num-
       ber of lines in its input (it  produces  no  output  other



Version 2.3                26 May 1990                          1





FLEX(1)                                                   FLEX(1)


       than  the  final  report  on  the counts).  The first line
       declares two globals, "num_lines" and  "num_chars",  which
       are  accessible both inside yyyylleexx(()) and in the mmaaiinn(()) rou-
       tine declared after the second "%%".  There are two rules,
       one which matches a newline ("\n") and increments both the
       line count and the character count, and one which  matches
       any  character  other than a newline (indicated by the "."
       regular expression).

       A somewhat more complicated example:

           /* scanner for a toy Pascal-like language */

           %{
           /* need this for the call to atof() below */
           #include <math.h>
           %}

           DIGIT    [0-9]
           ID       [a-z][a-z0-9]*

           %%

           {DIGIT}+    {
                       printf( "An integer: %s (%d)\n", yytext,
                               atoi( yytext ) );
                       }

           {DIGIT}+"."{DIGIT}*        {
                       printf( "A float: %s (%g)\n", yytext,
                               atof( yytext ) );
                       }

           if|then|begin|end|procedure|function        {
                       printf( "A keyword: %s\n", yytext );
                       }

           {ID}        printf( "An identifier: %s\n", yytext );

           "+"|"-"|"*"|"/"   printf( "An operator: %s\n", yytext );

           "{"[^}\n]*"}"     /* eat up one-line comments */

           [ \t\n]+          /* eat up whitespace */

           .           printf( "Unrecognized character: %s\n", yytext );

           %%

           main( argc, argv )
           int argc;
           char **argv;
               {
               ++argv, --argc;  /* skip over program name */



Version 2.3                26 May 1990                          2





FLEX(1)                                                   FLEX(1)


               if ( argc > 0 )
                       yyin = fopen( argv[0], "r" );
               else
                       yyin = stdin;

               yylex();
               }

       This is the beginnings of a simple scanner for a  language
       like  Pascal.  It identifies different types of _t_o_k_e_n_s and
       reports on what it has seen.

       The details of this example will be explained in the  fol-
       lowing sections.

FFOORRMMAATT OOFF TTHHEE IINNPPUUTT FFIILLEE
       The  _f_l_e_x input file consists of three sections, separated
       by a line with just %%%% in it:

           definitions
           %%
           rules
           %%
           user code

       The _d_e_f_i_n_i_t_i_o_n_s section contains  declarations  of  simple
       _n_a_m_e  definitions  to  simplify the scanner specification,
       and declarations of _s_t_a_r_t _c_o_n_d_i_t_i_o_n_s_, which are  explained
       in a later section.

       Name definitions have the form:

           name definition

       The  "name" is a word beginning with a letter or an under-
       score ('_') followed by zero or more letters, digits, '_',
       or  '-'  (dash).   The definition is taken to begin at the
       first non-white-space character  following  the  name  and
       continuing  to  the  end  of the line.  The definition can
       subsequently be referred to  using  "{name}",  which  will
       expand to "(definition)".  For example,

           DIGIT    [0-9]
           ID       [a-z][a-z0-9]*

       defines "DIGIT" to be a regular expression which matches a
       single digit, and "ID" to be a  regular  expression  which
       matches  a  letter  followed  by  zero-or-more letters-or-
       digits.  A subsequent reference to

           {DIGIT}+"."{DIGIT}*

       is identical to




Version 2.3                26 May 1990                          3





FLEX(1)                                                   FLEX(1)


           ([0-9])+"."([0-9])*

       and matches one-or-more digits followed by a '.'  followed
       by zero-or-more digits.

       The  _r_u_l_e_s  section of the _f_l_e_x input contains a series of
       rules of the form:

           pattern   action

       where the pattern must be unindented and the  action  must
       begin on the same line.

       See  below  for  a  further  description  of  patterns and
       actions.

       Finally,  the  user  code  section  is  simply  copied  to
       lleexx..yyyy..cc  verbatim.   It  is  used  for companion routines
       which call or are called by the scanner.  The presence  of
       this  section is optional; if it is missing, the second %%%%
       in the input file may be skipped, too.

       In the definitions and rules sections, any  _i_n_d_e_n_t_e_d  text
       or  text  enclosed  in %%{{ and %%}} is copied verbatim to the
       output (with the %{}'s removed).  The  %{}'s  must  appear
       unindented on lines by themselves.

       In  the  rules section, any indented or %{} text appearing
       before the first rule may be  used  to  declare  variables
       which  are  local  to  the scanning routine and (after the
       declarations) code which is to be  executed  whenever  the
       scanning  routine  is entered.  Other indented or %{} text
       in the rule section is still copied to the output, but its
       meaning is not well-defined and it may well cause compile-
       time errors (this feature is present for _P_O_S_I_X compliance;
       see below for other such features).

       In the definitions section, an unindented comment (i.e., a
       line beginning with "/*") is also copied verbatim  to  the
       output up to the next "*/".  Also, any line in the defini-
       tions section beginning with '#' is ignored,  though  this
       style  of  comment  is  deprecated  and may go away in the
       future.

PPAATTTTEERRNNSS
       The patterns in the input are written  using  an  extended
       set of regular expressions.  These are:

           x          match the character 'x'
           .          any character except newline
           [xyz]      a "character class"; in this case, the pattern
                        matches either an 'x', a 'y', or a 'z'
           [abj-oZ]   a "character class" with a range in it; matches
                        an 'a', a 'b', any letter from 'j' through 'o',



Version 2.3                26 May 1990                          4





FLEX(1)                                                   FLEX(1)


                        or a 'Z'
           [^A-Z]     a "negated character class", i.e., any character
                        but those in the class.  In this case, any
                        character EXCEPT an uppercase letter.
           [^A-Z\n]   any character EXCEPT an uppercase letter or
                        a newline
           r*         zero or more r's, where r is any regular expression
           r+         one or more r's
           r?         zero or one r's (that is, "an optional r")
           r{2,5}     anywhere from two to five r's
           r{2,}      two or more r's
           r{4}       exactly 4 r's
           {name}     the expansion of the "name" definition
                      (see above)
           "[xyz]\"foo"
                      the literal string: [xyz]"foo
           \X         if X is an 'a', 'b', 'f', 'n', 'r', 't', or 'v',
                        then the ANSI-C interpretation of \x.
                        Otherwise, a literal 'X' (used to escape
                        operators such as '*')
           \123       the character with octal value 123
           \x2a       the character with hexadecimal value 2a
           (r)        match an r; parentheses are used to override
                        precedence (see below)


           rs         the regular expression r followed by the
                        regular expression s; called "concatenation"


           r|s        either an r or an s


           r/s        an r but only if it is followed by an s.  The
                        s is not part of the matched text.  This type
                        of pattern is called as "trailing context".
           ^r         an r, but only at the beginning of a line
           r$         an r, but only at the end of a line.  Equivalent
                        to "r/\n".


           <s>r       an r, but only in start condition s (see
                      below for discussion of start conditions)
           <s1,s2,s3>r
                      same, but in any of start conditions s1,
                      s2, or s3


           <<EOF>>    an end-of-file
           <s1,s2><<EOF>>
                      an end-of-file when in start condition s1 or s2

       The regular expressions listed above are grouped according
       to precedence, from  highest  precedence  at  the  top  to



Version 2.3                26 May 1990                          5





FLEX(1)                                                   FLEX(1)


       lowest  at  the bottom.  Those grouped together have equal
       precedence.  For example,

           foo|bar*

       is the same as

           (foo)|(ba(r*))

       since the '*' operator has higher precedence than concate-
       nation,  and  concatenation higher than alternation ('|').
       This pattern therefore matches _e_i_t_h_e_r the string "foo"  _o_r
       the  string  "ba"  followed by zero-or-more r's.  To match
       "foo" or zero-or-more "bar"'s, use:

           foo|(bar)*

       and to match zero-or-more "foo"'s-or-"bar"'s:

           (foo|bar)*


       Some notes on patterns:

       -      A negated character class such as the example "[^A-
              Z]"  above  _w_i_l_l _m_a_t_c_h _a _n_e_w_l_i_n_e unless "\n" (or an
              equivalent escape sequence) is one of  the  charac-
              ters  explicitly  present  in the negated character
              class (e.g., "[^A-Z\n]").  This is unlike how  many
              other  regular expression tools treat negated char-
              acter classes, but unfortunately the  inconsistency
              is   historically  entrenched.   Matching  newlines
              means that a pattern like [^"]* can match an entire
              input  (overflowing  the  scanner's  input  buffer)
              unless there's another quote in the input.

       -      A rule can have at most one  instance  of  trailing
              context  (the  '/'  operator  or the '$' operator).
              The start condition, '^',  and  "<<EOF>>"  patterns
              can  only occur at the beginning of a pattern, and,
              as well as with '/'  and  '$',  cannot  be  grouped
              inside  parentheses.  A '^' which does not occur at
              the beginning of a rule or a  '$'  which  does  not
              occur  at the end of a rule loses its special prop-
              erties and is treated as a normal character.

              The following are illegal:

                  foo/bar$
                  <sc1>foo<sc2>bar

              Note that  the  first  of  these,  can  be  written
              "foo/bar\n".




Version 2.3                26 May 1990                          6





FLEX(1)                                                   FLEX(1)


              The  following  will  result  in  '$'  or '^' being
              treated as a normal character:

                  foo|(bar$)
                  foo|^bar

              If what's wanted is a "foo" or a bar-followed-by-a-
              newline,  the  following could be used (the special
              '|' action is explained below):

                  foo      |
                  bar$     /* action goes here */

              A similar trick will work for matching a foo  or  a
              bar-at-the-beginning-of-a-line.

HHOOWW TTHHEE IINNPPUUTT IISS MMAATTCCHHEEDD
       When  the  generated scanner is run, it analyzes its input
       looking for strings which match any of its  patterns.   If
       it  finds  more  than one match, it takes the one matching
       the most text (for trailing context rules,  this  includes
       the  length of the trailing part, even though it will then
       be returned to the  input).   If  it  finds  two  or  more
       matches  of  the same length, the rule listed first in the
       _f_l_e_x input file is chosen.

       Once the match is determined, the  text  corresponding  to
       the  match  (called  the  _t_o_k_e_n_)  is made available in the
       global character pointer yyyytteexxtt,, and  its  length  in  the
       global  integer  yyyylleenngg..   The _a_c_t_i_o_n corresponding to the
       matched pattern is then executed (a more detailed descrip-
       tion  of actions follows), and then the remaining input is
       scanned for another match.

       If no match is found, then the _d_e_f_a_u_l_t _r_u_l_e  is  executed:
       the  next character in the input is considered matched and
       copied to the standard output.  Thus, the  simplest  legal
       _f_l_e_x input is:

           %%

       which  generates  a  scanner  that simply copies its input
       (one character at a time) to its output.

AACCTTIIOONNSS
       Each pattern in a rule has a corresponding  action,  which
       can be any arbitrary C statement.  The pattern ends at the
       first non-escaped whitespace character; the  remainder  of
       the line is its action.  If the action is empty, then when
       the pattern is matched the  input  token  is  simply  dis-
       carded.  For example, here is the specification for a pro-
       gram which deletes all occurrences of "zap  me"  from  its
       input:




Version 2.3                26 May 1990                          7





FLEX(1)                                                   FLEX(1)


           %%
           "zap me"

       (It  will  copy  all  other characters in the input to the
       output since they will be matched by the default rule.)

       Here is a program which  compresses  multiple  blanks  and
       tabs  down  to  a single blank, and throws away whitespace
       found at the end of a line:

           %%
           [ \t]+        putchar( ' ' );
           [ \t]+$       /* ignore this token */


       If the action contains a '{', then the action  spans  till
       the  balancing '}' is found, and the action may cross mul-
       tiple lines.  _f_l_e_x knows about C strings and comments  and
       won't  be  fooled  by  braces  found within them, but also
       allows actions to begin with  %%{{  and  will  consider  the
       action to be all the text up to the next %%}} (regardless of
       ordinary braces inside the action).

       An action consisting solely of a vertical bar ('|')  means
       "same  as the action for the next rule."  See below for an
       illustration.

       Actions can include arbitrary  C  code,  including  rreettuurrnn
       statements  to  return  a value to whatever routine called
       yyyylleexx(())..  Each time yyyylleexx(()) is called  it  continues  pro-
       cessing tokens from where it last left off until it either
       reaches the end of the file or executes a return.  Once it
       reaches  an end-of-file, however, then any subsequent call
       to  yyyylleexx(())  will  simply   immediately   return,   unless
       yyyyrreessttaarrtt(()) is first called (see below).

       Actions are not allowed to modify yytext or yyleng.

       There  are  a  number  of  special directives which can be
       included within an action:

       -      EECCHHOO copies yytext to the scanner's output.

       -      BBEEGGIINN followed by the name  of  a  start  condition
              places  the scanner in the corresponding start con-
              dition (see below).

       -      RREEJJEECCTT directs the scanner to  proceed  on  to  the
              "second  best"  rule  which matched the input (or a
              prefix of  the  input).   The  rule  is  chosen  as
              described  above in "How the Input is Matched", and
              yyyytteexxtt and yyyylleenngg set  up  appropriately.   It  may
              either  be  one  which  matched as much text as the
              originally chosen rule but came later in  the  _f_l_e_x



Version 2.3                26 May 1990                          8





FLEX(1)                                                   FLEX(1)


              input  file,  or  one which matched less text.  For
              example, the following will both count the words in
              the  input  and call the routine special() whenever
              "frob" is seen:

                          int word_count = 0;
                  %%

                  frob        special(); REJECT;
                  [^ \t\n]+   ++word_count;

              Without the RREEJJEECCTT,, any "frob"'s in the input would
              not be counted as words, since the scanner normally
              executes  only  one  action  per  token.   Multiple
              RREEJJEECCTT''ss  are  allowed,  each  one finding the next
              best choice to  the  currently  active  rule.   For
              example, when the following scanner scans the token
              "abcd", it will write "abcdabcaba" to the output:

                  %%
                  a        |
                  ab       |
                  abc      |
                  abcd     ECHO; REJECT;
                  .|\n     /* eat up any unmatched character */

              (The first three rules share  the  fourth's  action
              since  they use the special '|' action.)  RREEJJEECCTT is
              a particularly expensive feature in  terms  scanner
              performance;  if it is used in _a_n_y of the scanner's
              actions it will slow  down  _a_l_l  of  the  scanner's
              matching.   Furthermore, RREEJJEECCTT cannot be used with
              the _-_f or _-_F options (see below).

              Note also that unlike the  other  special  actions,
              RREEJJEECCTT  is  a _b_r_a_n_c_h_; code immediately following it
              in the action will _n_o_t be executed.

       -      yyyymmoorree(()) tells the scanner that the  next  time  it
              matches  a  rule, the corresponding token should be
              _a_p_p_e_n_d_e_d onto the current value  of  yyyytteexxtt  rather
              than  replacing  it.   For example, given the input
              "mega-kludge" the following will write  "mega-mega-
              kludge" to the output:

                  %%
                  mega-    ECHO; yymore();
                  kludge   ECHO;

              First  "mega-" is matched and echoed to the output.
              Then "kludge" is matched, but the previous  "mega-"
              is  still hanging around at the beginning of yyyytteexxtt
              so the EECCHHOO for the  "kludge"  rule  will  actually
              write  "mega-kludge".   The presence of yyyymmoorree(()) in



Version 2.3                26 May 1990                          9





FLEX(1)                                                   FLEX(1)


              the scanner's action entails  a  minor  performance
              penalty in the scanner's matching speed.

       -      yyyylleessss((nn)) returns all but the first _n characters of
              the current token back to the input  stream,  where
              they  will  be rescanned when the scanner looks for
              the next match.  yyyytteexxtt  and  yyyylleenngg  are  adjusted
              appropriately  (e.g., yyyylleenngg will now be equal to _n
              ).  For example, on the input "foobar" the  follow-
              ing will write out "foobarbar":

                  %%
                  foobar    ECHO; yyless(3);
                  [a-z]+    ECHO;

              An  argument  of  0 to yyyylleessss will cause the entire
              current input string to be scanned  again.   Unless
              you've  changed  how  the scanner will subsequently
              process its input (using BBEEGGIINN,, for example),  this
              will result in an endless loop.

       -      uunnppuutt((cc))  puts  the character _c back onto the input
              stream.  It will be  the  next  character  scanned.
              The  following  action  will take the current token
              and cause it to be rescanned enclosed in  parenthe-
              ses.

                  {
                  int i;
                  unput( ')' );
                  for ( i = yyleng - 1; i >= 0; --i )
                      unput( yytext[i] );
                  unput( '(' );
                  }

              Note that since each uunnppuutt(()) puts the given charac-
              ter back at the  _b_e_g_i_n_n_i_n_g  of  the  input  stream,
              pushing back strings must be done back-to-front.

       -      iinnppuutt(())  reads  the  next  character from the input
              stream.  For example, the following is one  way  to
              eat up C comments:

                  %%
                  "/*"        {
                              register int c;

                              for ( ; ; )
                                  {
                                  while ( (c = input()) != '*' &&
                                          c != EOF )
                                      ;    /* eat up text of comment */

                                  if ( c == '*' )



Version 2.3                26 May 1990                         10





FLEX(1)                                                   FLEX(1)


                                      {
                                      while ( (c = input()) == '*' )
                                          ;
                                      if ( c == '/' )
                                          break;    /* found the end */
                                      }

                                  if ( c == EOF )
                                      {
                                      error( "EOF in comment" );
                                      break;
                                      }
                                  }
                              }

              (Note  that  if  the scanner is compiled using CC++++,,
              then iinnppuutt(()) is instead referred to  as  yyyyiinnppuutt(()),,
              in  order to avoid a name clash with the CC++++ stream
              by the name of _i_n_p_u_t_._)

       -      yyyytteerrmmiinnaattee(()) can be  used  in  lieu  of  a  return
              statement  in an action.  It terminates the scanner
              and returns a 0 to the scanner's caller, indicating
              "all  done".   Subsequent calls to the scanner will
              immediately return unless preceded  by  a  call  to
              yyyyrreessttaarrtt(()) (see below).  By default, yyyytteerrmmiinnaattee(())
              is also called when an end-of-file is  encountered.
              It is a macro and may be redefined.

TTHHEE GGEENNEERRAATTEEDD SSCCAANNNNEERR
       The  output  of  _f_l_e_x is the file lleexx..yyyy..cc,, which contains
       the scanning routine yyyylleexx(()),, a number of tables  used  by
       it for matching tokens, and a number of auxiliary routines
       and macros.  By default, yyyylleexx(()) is declared as follows:

           int yylex()
               {
               ... various definitions and the actions in here ...
               }

       (If your environment supports function prototypes, then it
       will  be  "int  yylex(  void  )".)  This definition may be
       changed by redefining the "YY_DECL" macro.   For  example,
       you could use:

           #undef YY_DECL
           #define YY_DECL float lexscan( a, b ) float a, b;

       to give the scanning routine the name _l_e_x_s_c_a_n_, returning a
       float, and taking two floats as arguments.  Note  that  if
       you  give  arguments  to the scanning routine using a K&R-
       style/non-prototyped function declaration, you must termi-
       nate the definition with a semi-colon (;).




Version 2.3                26 May 1990                         11





FLEX(1)                                                   FLEX(1)


       Whenever  yyyylleexx(())  is  called,  it  scans  tokens from the
       global input file _y_y_i_n (which defaults to stdin).  It con-
       tinues  until  it  either reaches an end-of-file (at which
       point it returns the value 0) or one of its  actions  exe-
       cutes a _r_e_t_u_r_n statement.  In the former case, when called
       again  the  scanner   will   immediately   return   unless
       yyyyrreessttaarrtt(()) is called to point _y_y_i_n at the new input file.
       ( yyyyrreessttaarrtt(()) takes one argument, a FFIILLEE **  pointer.)   In
       the  latter case (i.e., when an action executes a return),
       the scanner may then be called again and  it  will  resume
       scanning where it left off.

       By  default  (and for purposes of efficiency), the scanner
       uses block-reads rather than simple _g_e_t_c_(_) calls  to  read
       characters from _y_y_i_n_.  The nature of how it gets its input
       can  be  controlled  by  redefining  the  YYYY__IINNPPUUTT  macro.
       YY_INPUT's           calling          sequence          is
       "YY_INPUT(buf,result,max_size)".  Its action is  to  place
       up  to  _m_a_x___s_i_z_e characters in the character array _b_u_f and
       return in the integer variable _r_e_s_u_l_t either the number of
       characters  read  or  the constant YY_NULL (0 on Unix sys-
       tems) to indicate EOF.  The default  YY_INPUT  reads  from
       the global file-pointer "yyin".

       A sample redefinition of YY_INPUT (in the definitions sec-
       tion of the input file):

           %{
           #undef YY_INPUT
           #define YY_INPUT(buf,result,max_size) \
               { \
               int c = getchar(); \
               result = (c == EOF) ? YY_NULL : (buf[0] = c, 1); \
               }
           %}

       This definition will change the input processing to  occur
       one character at a time.

       You also can add in things like keeping track of the input
       line number this way; but don't expect your scanner to  go
       very fast.

       When  the  scanner receives an end-of-file indication from
       YY_INPUT,  it  then  checks  the  yyyywwrraapp(())  function.   If
       yyyywwrraapp(()) returns false (zero), then it is assumed that the
       function has gone ahead  and  set  up  _y_y_i_n  to  point  to
       another input file, and scanning continues.  If it returns
       true (non-zero), then the scanner terminates, returning  0
       to its caller.

       The  default  yyyywwrraapp(())  always  returns  1.  Presently, to
       redefine it you must first "#undef yywrap", as it is  cur-
       rently  implemented  as  a  macro.   As  indicated  by the



Version 2.3                26 May 1990                         12





FLEX(1)                                                   FLEX(1)


       hedging in the previous sentence, it may be changed  to  a
       true function in the near future.

       The  scanner  writes  its  EECCHHOO output to the _y_y_o_u_t global
       (default, stdout), which may be redefined by the user sim-
       ply by assigning it to some other FFIILLEE pointer.

SSTTAARRTT CCOONNDDIITTIIOONNSS
       _f_l_e_x  provides  a  mechanism  for conditionally activating
       rules.  Any rule whose pattern  is  prefixed  with  "<sc>"
       will  only be active when the scanner is in the start con-
       dition named "sc".  For example,

           <STRING>[^"]*        { /* eat up the string body ... */
                       ...
                       }

       will be active only when the scanner is  in  the  "STRING"
       start condition, and

           <INITIAL,STRING,QUOTE>\.        { /* handle an escape ... */
                       ...
                       }

       will  be  active  only when the current start condition is
       either "INITIAL", "STRING", or "QUOTE".

       Start conditions are declared in the  definitions  (first)
       section of the input using unindented lines beginning with
       either %%ss or %%xx followed by a list of names.   The  former
       declares  _i_n_c_l_u_s_i_v_e start conditions, the latter _e_x_c_l_u_s_i_v_e
       start conditions.  A start condition  is  activated  using
       the  BBEEGGIINN  action.   Until  the next BBEEGGIINN action is exe-
       cuted, rules with the given start condition will be active
       and  rules  with  other start conditions will be inactive.
       If the start condition is _i_n_c_l_u_s_i_v_e_, then  rules  with  no
       start  conditions  at  all  will also be active.  If it is
       _e_x_c_l_u_s_i_v_e_, then _o_n_l_y rules qualified with the start condi-
       tion  will  be  active.   A set of rules contingent on the
       same exclusive start condition describe a scanner which is
       independent  of  any of the other rules in the _f_l_e_x input.
       Because of this, exclusive start conditions make  it  easy
       to  specify  "mini-scanners"  which  scan  portions of the
       input that  are  syntactically  different  from  the  rest
       (e.g., comments).

       If  the  distinction between inclusive and exclusive start
       conditions is still a little vague, here's a simple  exam-
       ple  illustrating the connection between the two.  The set
       of rules:

           %s example
           %%
           <example>foo           /* do something */



Version 2.3                26 May 1990                         13





FLEX(1)                                                   FLEX(1)


       is equivalent to

           %x example
           %%
           <INITIAL,example>foo   /* do something */


       The default rule (to EECCHHOO any unmatched character) remains
       active in start conditions.

       BBEEGGIINN((00))  returns  to  the  original  state where only the
       rules with no start conditions are active.  This state can
       also  be  referred to as the start-condition "INITIAL", so
       BBEEGGIINN((IINNIITTIIAALL)) is equivalent to BBEEGGIINN((00))..  (The  parenthe-
       ses  around  the start condition name are not required but
       are considered good style.)

       BBEEGGIINN actions can also be given as indented  code  at  the
       beginning  of the rules section.  For example, the follow-
       ing will cause the scanner to enter  the  "SPECIAL"  start
       condition  whenever _y_y_l_e_x_(_) is called and the global vari-
       able _e_n_t_e_r___s_p_e_c_i_a_l is true:

                   int enter_special;

           %x SPECIAL
           %%
                   if ( enter_special )
                       BEGIN(SPECIAL);

           <SPECIAL>blahblahblah
           ...more rules follow...


       To illustrate the uses of  start  conditions,  here  is  a
       scanner  which provides two different interpretations of a
       string like "123.456".  By default it will treat it as  as
       three  tokens,  the  integer  "123",  a dot ('.'), and the
       integer "456".  But if the string is preceded  earlier  in
       the line by the string "expect-floats" it will treat it as
       a single token, the floating-point number 123.456:

           %{
           #include <math.h>
           %}
           %s expect

           %%
           expect-floats        BEGIN(expect);

           <expect>[0-9]+"."[0-9]+      {
                       printf( "found a float, = %f\n",
                               atof( yytext ) );
                       }



Version 2.3                26 May 1990                         14





FLEX(1)                                                   FLEX(1)


           <expect>\n           {
                       /* that's the end of the line, so
                        * we need another "expect-number"
                        * before we'll recognize any more
                        * numbers
                        */
                       BEGIN(INITIAL);
                       }

           [0-9]+      {
                       printf( "found an integer, = %d\n",
                               atoi( yytext ) );
                       }

           "."         printf( "found a dot\n" );

       Here is a scanner which recognizes (and discards)  C  com-
       ments while maintaining a count of the current input line.

           %x comment
           %%
                   int line_num = 1;

           "/*"         BEGIN(comment);

           <comment>[^*\n]*        /* eat anything that's not a '*' */
           <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
           <comment>\n             ++line_num;
           <comment>"*"+"/"        BEGIN(INITIAL);

       Note that start-conditions names are really integer values
       and  can  be  stored  as  such.   Thus, the above could be
       extended in the following fashion:

           %x comment foo
           %%
                   int line_num = 1;
                   int comment_caller;

           "/*"         {
                        comment_caller = INITIAL;
                        BEGIN(comment);
                        }

           ...

           <foo>"/*"    {
                        comment_caller = foo;
                        BEGIN(comment);
                        }

           <comment>[^*\n]*        /* eat anything that's not a '*' */
           <comment>"*"+[^*/\n]*   /* eat up '*'s not followed by '/'s */
           <comment>\n             ++line_num;



Version 2.3                26 May 1990                         15





FLEX(1)                                                   FLEX(1)


           <comment>"*"+"/"        BEGIN(comment_caller);

       One can then implement a "stack" of start conditions using
       an array of integers.  (It is likely that such stacks will
       become a full-fledged _f_l_e_x feature in the future.)   Note,
       though,  that start conditions do not have their own name-
       space; %s's and %x's declare names in the same fashion  as
       #define's.

MMUULLTTIIPPLLEE IINNPPUUTT BBUUFFFFEERRSS
       Some  scanners  (such  as  those  which  support "include"
       files) require reading from  several  input  streams.   As
       _f_l_e_x  scanners  do a large amount of buffering, one cannot
       control where the next input will be read from  by  simply
       writing a YYYY__IINNPPUUTT which is sensitive to the scanning con-
       text.  YYYY__IINNPPUUTT is only called when  the  scanner  reaches
       the  end  of  its  buffer,  which may be a long time after
       scanning a statement such as an "include"  which  requires
       switching the input source.

       To  negotiate  these  sorts  of  problems, _f_l_e_x provides a
       mechanism for  creating  and  switching  between  multiple
       input buffers.  An input buffer is created by using:

           YY_BUFFER_STATE yy_create_buffer( FILE *file, int size )

       which takes a _F_I_L_E pointer and a size and creates a buffer
       associated with the given file and large  enough  to  hold
       _s_i_z_e  characters  (when  in doubt, use YYYY__BBUUFF__SSIIZZEE for the
       size).  It returns a  YYYY__BBUUFFFFEERR__SSTTAATTEE  handle,  which  may
       then be passed to other routines:

           void yy_switch_to_buffer( YY_BUFFER_STATE new_buffer )

       switches  the  scanner's input buffer so subsequent tokens
       will     come     from     _n_e_w___b_u_f_f_e_r_.      Note      that
       yyyy__sswwiittcchh__ttoo__bbuuffffeerr(())  may  be  used  by  yywrap() to sets
       things up for continued scanning, instead of opening a new
       file and pointing _y_y_i_n at it.

           void yy_delete_buffer( YY_BUFFER_STATE buffer )

       is used to reclaim the storage associated with a buffer.

       yyyy__nneeww__bbuuffffeerr(())  is  an alias for yyyy__ccrreeaattee__bbuuffffeerr(()),, pro-
       vided for compatibility with the C++ use of _n_e_w and _d_e_l_e_t_e
       for creating and destroying dynamic objects.

       Finally,    the    YYYY__CCUURRRREENNTT__BBUUFFFFEERR   macro   returns   a
       YYYY__BBUUFFFFEERR__SSTTAATTEE handle to the current buffer.

       Here is an example of using these features for  writing  a
       scanner  which  expands include files (the <<<<EEOOFF>>>> feature
       is discussed below):



Version 2.3                26 May 1990                         16





FLEX(1)                                                   FLEX(1)


           /* the "incl" state is used for picking up the name
            * of an include file
            */
           %x incl

           %{
           #define MAX_INCLUDE_DEPTH 10
           YY_BUFFER_STATE include_stack[MAX_INCLUDE_DEPTH];
           int include_stack_ptr = 0;
           %}

           %%
           include             BEGIN(incl);

           [a-z]+              ECHO;
           [^a-z\n]*\n?        ECHO;

           <incl>[ \t]*      /* eat the whitespace */
           <incl>[^ \t\n]+   { /* got the include file name */
                   if ( include_stack_ptr >= MAX_INCLUDE_DEPTH )
                       {
                       fprintf( stderr, "Includes nested too deeply" );
                       exit( 1 );
                       }

                   include_stack[include_stack_ptr++] =
                       YY_CURRENT_BUFFER;

                   yyin = fopen( yytext, "r" );

                   if ( ! yyin )
                       error( ... );

                   yy_switch_to_buffer(
                       yy_create_buffer( yyin, YY_BUF_SIZE ) );

                   BEGIN(INITIAL);
                   }

           <<EOF>> {
                   if ( --include_stack_ptr < 0 )
                       {
                       yyterminate();
                       }

                   else
                       yy_switch_to_buffer(
                            include_stack[include_stack_ptr] );
                   }


EENNDD--OOFF--FFIILLEE RRUULLEESS
       The special rule "<<EOF>>" indicates actions which are  to
       be  taken  when an end-of-file is encountered and yywrap()



Version 2.3                26 May 1990                         17





FLEX(1)                                                   FLEX(1)


       returns non-zero (i.e., indicates no further files to pro-
       cess).   The  action  must  finish  by  doing  one of four
       things:

       -      the special YYYY__NNEEWW__FFIILLEE action, if  _y_y_i_n  has  been
              pointed at a new file to process;

       -      a _r_e_t_u_r_n statement;

       -      the special yyyytteerrmmiinnaattee(()) action;

       -      or,    switching    to    a    new   buffer   using
              yyyy__sswwiittcchh__ttoo__bbuuffffeerr(())  as  shown  in  the   example
              above.

       <<EOF>>  rules  may  not be used with other patterns; they
       may only be qualified with a list of start conditions.  If
       an  unqualified  <<EOF>>  rule is given, it applies to _a_l_l
       start  conditions  which  do  not  already  have   <<EOF>>
       actions.   To specify an <<EOF>> rule for only the initial
       start condition, use

           <INITIAL><<EOF>>


       These rules are useful for catching things  like  unclosed
       comments.  An example:

           %x quote
           %%

           ...other rules for dealing with quotes...

           <quote><<EOF>>   {
                    error( "unterminated quote" );
                    yyterminate();
                    }
           <<EOF>>  {
                    if ( *++filelist )
                        {
                        yyin = fopen( *filelist, "r" );
                        YY_NEW_FILE;
                        }
                    else
                       yyterminate();
                    }


MMIISSCCEELLLLAANNEEOOUUSS MMAACCRROOSS
       The  macro  YY_USER_ACTION  can be redefined to provide an
       action which is  always  executed  prior  to  the  matched
       rule's action.  For example, it could be #define'd to call
       a routine to convert yytext to lower-case.




Version 2.3                26 May 1990                         18





FLEX(1)                                                   FLEX(1)


       The macro YYYY__UUSSEERR__IINNIITT may  be  redefined  to  provide  an
       action which is always executed before the first scan (and
       before the scanner's internal initializations  are  done).
       For example, it could be used to call a routine to read in
       a data table or open a logging file.

       In the generated scanner, the actions are all gathered  in
       one  large  switch statement and separated using YYYY__BBRREEAAKK,,
       which may be  redefined.   By  default,  it  is  simply  a
       "break", to separate each rule's action from the following
       rule's.  Redefining  YYYY__BBRREEAAKK  allows,  for  example,  C++
       users  to #define YY_BREAK to do nothing (while being very
       careful  that  every  rule  ends  with  a  "break"  or   a
       "return"!)  to  avoid suffering from unreachable statement
       warnings where because a rule's action ends with "return",
       the YYYY__BBRREEAAKK is inaccessible.

IINNTTEERRFFAACCIINNGG WWIITTHH YYAACCCC
       One of the main uses of _f_l_e_x is as a companion to the _y_a_c_c
       parser-generator.  _y_a_c_c parsers expect to call  a  routine
       named  yyyylleexx(())  to find the next input token.  The routine
       is supposed to return the type of the next token  as  well
       as  putting any associated value in the global yyyyllvvaall..  To
       use _f_l_e_x with _y_a_c_c_, one specifies the --dd option to _y_a_c_c to
       instruct  it to generate the file yy..ttaabb..hh containing defi-
       nitions of all the %%ttookkeennss appearing in  the  _y_a_c_c  input.
       This file is then included in the _f_l_e_x scanner.  For exam-
       ple, if one of the tokens is  "TOK_NUMBER",  part  of  the
       scanner might look like:

           %{
           #include "y.tab.h"
           %}

           %%

           [0-9]+        yylval = atoi( yytext ); return TOK_NUMBER;


TTRRAANNSSLLAATTIIOONN TTAABBLLEE
       In  the name of POSIX compliance, _f_l_e_x supports a _t_r_a_n_s_l_a_-
       _t_i_o_n _t_a_b_l_e for mapping input characters into groups.   The
       table  is  specified  in the first section, and its format
       looks like:

           %t
           1        abcd
           2        ABCDEFGHIJKLMNOPQRSTUVWXYZ
           52       0123456789
           6        \t\ \n
           %t

       This example specifies that the characters 'a', 'b',  'c',
       and  'd'  are  to  all be lumped into group #1, upper-case



Version 2.3                26 May 1990                         19





FLEX(1)                                                   FLEX(1)


       letters in group #2, digits in group  #52,  tabs,  blanks,
       and  newlines  into group #6, and _n_o _o_t_h_e_r _c_h_a_r_a_c_t_e_r_s _w_i_l_l
       _a_p_p_e_a_r _i_n _t_h_e _p_a_t_t_e_r_n_s_.  The group  numbers  are  actually
       disregarded by _f_l_e_x_; %%tt serves, though, to lump characters
       together.  Given the above table, for example, the pattern
       "a(AA)*5"  is  equivalent  to  "d(ZQ)*0".   They both say,
       "match any character in group #1, followed by zero-or-more
       pairs of characters from group #2, followed by a character
       from group #52."  Thus %%tt provides a crude way for  intro-
       ducing equivalence classes into the scanner specification.

       Note that the --ii  option  (see  below)  coupled  with  the
       equivalence  classes  which  _f_l_e_x  automatically generates
       take care of virtually all the instances  when  one  might
       consider  using  %%tt..  But what the hell, it's there if you
       want it.

OOPPTTIIOONNSS
       _f_l_e_x has the following options:

       --bb     Generate backtracking information to _l_e_x_._b_a_c_k_t_r_a_c_k_.
              This  is  a  list  of  scanner states which require
              backtracking and the input characters on which they
              do so.  By adding rules one can remove backtracking
              states.  If all backtracking states are  eliminated
              and  --ff  or  --FF is used, the generated scanner will
              run faster (see the --pp flag).  Only users who  wish
              to  squeeze  every last cycle out of their scanners
              need worry about this option.  (See the section  on
              PERFORMANCE CONSIDERATIONS below.)

       --cc     is  a  do-nothing,  deprecated  option included for
              POSIX compliance.

              NNOOTTEE:: in previous releases  of  _f_l_e_x  --cc  specified
              table-compression  options.   This functionality is
              now given by the --CC flag.  To ease the  the  impact
              of  this  change,  when _f_l_e_x encounters --cc,, it cur-
              rently issues a warning message and assumes that --CC
              was  desired  instead.   In the future this "promo-
              tion" of --cc to --CC will go away in the name of  full
              POSIX  compliance  (unless  the  POSIX  meaning  is
              removed first).

       --dd     makes the generated  scanner  run  in  _d_e_b_u_g  mode.
              Whenever  a  pattern  is  recognized and the global
              yyyy__fflleexx__ddeebbuugg is non-zero (which is  the  default),
              the  scanner  will  write  to  _s_t_d_e_r_r a line of the
              form:

                  --accepting rule at line 53 ("the matched text")

              The line number refers to the location of the  rule
              in  the  file  defining the scanner (i.e., the file



Version 2.3                26 May 1990                         20





FLEX(1)                                                   FLEX(1)


              that was fed to flex).  Messages are also generated
              when  the  scanner  backtracks, accepts the default
              rule, reaches the  end  of  its  input  buffer  (or
              encounters  a  NUL; at this point, the two look the
              same as far as the scanner's concerned), or reaches
              an end-of-file.

       --ff     specifies (take your pick) _f_u_l_l _t_a_b_l_e or _f_a_s_t _s_c_a_n_-
              _n_e_r_.  No table compression is done.  The result  is
              large  but  fast.  This option is equivalent to --CCff
              (see below).

       --ii     instructs _f_l_e_x to generate a _c_a_s_e_-_i_n_s_e_n_s_i_t_i_v_e scan-
              ner.   The  case of letters given in the _f_l_e_x input
              patterns will be ignored, and tokens in  the  input
              will  be  matched  regardless of case.  The matched
              text given in _y_y_t_e_x_t will have the  preserved  case
              (i.e., it will not be folded).

       --nn     is  another  do-nothing, deprecated option included
              only for POSIX compliance.

       --pp     generates a  performance  report  to  stderr.   The
              report  consists  of comments regarding features of
              the _f_l_e_x input file which will cause a loss of per-
              formance  in  the resulting scanner.  Note that the
              use of _R_E_J_E_C_T and variable  trailing  context  (see
              the  BUGS section in flex(1)) entails a substantial
              performance penalty; use of _y_y_m_o_r_e_(_)_, the ^^  opera-
              tor,  and  the  --II  flag  entail  minor performance
              penalties.

       --ss     causes the _d_e_f_a_u_l_t  _r_u_l_e  (that  unmatched  scanner
              input  is  echoed  to _s_t_d_o_u_t_) to be suppressed.  If
              the scanner encounters input that  does  not  match
              any  of  its  rules, it aborts with an error.  This
              option is useful for finding holes in  a  scanner's
              rule set.

       --tt     instructs _f_l_e_x to write the scanner it generates to
              standard output instead of lleexx..yyyy..cc..

       --vv     specifies that _f_l_e_x should write to _s_t_d_e_r_r  a  sum-
              mary  of statistics regarding the scanner it gener-
              ates.  Most of the statistics  are  meaningless  to
              the casual _f_l_e_x user, but the first line identifies
              the version of _f_l_e_x_, which is useful  for  figuring
              out where you stand with respect to patches and new
              releases, and the next two lines give the date when
              the  scanner was created and a summary of the flags
              which were in effect.

       --FF     specifies that the _f_a_s_t scanner  table  representa-
              tion  should be used.  This representation is about



Version 2.3                26 May 1990                         21





FLEX(1)                                                   FLEX(1)


              as fast as the full table representation _(_-_f_)_,  and
              for  some  sets  of  patterns  will be considerably
              smaller (and for others, larger).  In  general,  if
              the  pattern  set  contains  both  "keywords" and a
              catch-all, "identifier" rule, such as in the set:

                  "case"    return TOK_CASE;
                  "switch"  return TOK_SWITCH;
                  ...
                  "default" return TOK_DEFAULT;
                  [a-z]+    return TOK_ID;

              then you're better off using the full table  repre-
              sentation.   If  only the "identifier" rule is pre-
              sent and you then use a hash table or some such  to
              detect the keywords, you're better off using _-_F_.

              This option is equivalent to --CCFF (see below).

       --II     instructs  _f_l_e_x to generate an _i_n_t_e_r_a_c_t_i_v_e scanner.
              Normally, scanners generated by  _f_l_e_x  always  look
              ahead one character before deciding that a rule has
              been matched.  At the cost of some  scanning  over-
              head, _f_l_e_x will generate a scanner which only looks
              ahead when needed.  Such scanners are called _i_n_t_e_r_-
              _a_c_t_i_v_e  because  if you want to write a scanner for
              an interactive system such as a command shell,  you
              will  probably  want  the user's input to be termi-
              nated with a newline, and without --II the user  will
              have to type a character in addition to the newline
              in order to  have  the  newline  recognized.   This
              leads to dreadful interactive performance.

              If  all this seems to confusing, here's the general
              rule: if a human will be typing in  input  to  your
              scanner, use --II,, otherwise don't; if you don't care
              about squeezing the utmost  performance  from  your
              scanner  and you don't want to make any assumptions
              about the input to your scanner, use --II..

              Note, --II cannot be used in conjunction with _f_u_l_l or
              _f_a_s_t _t_a_b_l_e_s_, i.e., the --ff,, --FF,, --CCff,, or --CCFF flags.

       --LL     instructs  _f_l_e_x  not  to generate ##lliinnee directives.
              Without this option,  _f_l_e_x  peppers  the  generated
              scanner  with #line directives so error messages in
              the actions will be correctly located with  respect
              to  the  original  _f_l_e_x  input file, and not to the
              fairly  meaningless  line  numbers   of   lleexx..yyyy..cc..
              (Unfortunately _f_l_e_x does not presently generate the
              necessary directives to "retarget" the line numbers
              for those parts of lleexx..yyyy..cc which it generated.  So
              if there is an error in the generated code, a mean-
              ingless line number is reported.)



Version 2.3                26 May 1990                         22





FLEX(1)                                                   FLEX(1)


       --TT     makes  _f_l_e_x  run in _t_r_a_c_e mode.  It will generate a
              lot of messages to _s_t_d_o_u_t concerning  the  form  of
              the  input  and the resultant non-deterministic and
              deterministic  finite  automata.   This  option  is
              mostly for use in maintaining _f_l_e_x_.

       --88     instructs  _f_l_e_x to generate an 8-bit scanner, i.e.,
              one which can recognize 8-bit characters.  On  some
              sites,  _f_l_e_x  is  installed with this option as the
              default.  On others, the default is  7-bit  charac-
              ters.   To see which is the case, check the verbose
              ((--vv)) output for "equivalence classes created".   If
              the denominator of the number shown is 128, then by
              default _f_l_e_x is generating 7-bit characters.  If it
              is  256,  then  the default is 8-bit characters and
              the --88 flag is not required (but may be a good idea
              to keep the scanner specification portable).  Feed-
              ing a 7-bit scanner 8-bit characters will result in
              infinite  loops,  bus  errors,  or other such fire-
              works, so when in doubt, use the flag.   Note  that
              if  equivalence  classes  are  used, 8-bit scanners
              take only slightly  more  table  space  than  7-bit
              scanners  (128  bytes, to be exact); if equivalence
              classes are not used, however, then the tables  may
              grow up to twice their 7-bit size.

       --CC[[eeffmmFF]]
              controls the degree of table compression.

              --CCee  directs _f_l_e_x to construct _e_q_u_i_v_a_l_e_n_c_e _c_l_a_s_s_e_s_,
              i.e., sets of characters which have identical lexi-
              cal properties (for example, if the only appearance
              of digits in the _f_l_e_x input  is  in  the  character
              class  "[0-9]"  then  the digits '0', '1', ..., '9'
              will all be put in  the  same  equivalence  class).
              Equivalence  classes  usually  give dramatic reduc-
              tions in the final table/object file  sizes  (typi-
              cally a factor of 2-5) and are pretty cheap perfor-
              mance-wise  (one  array   look-up   per   character
              scanned).

              --CCff  specifies  that the _f_u_l_l scanner tables should
              be generated - _f_l_e_x should not compress the  tables
              by  taking  advantages  of similar transition func-
              tions for different states.

              --CCFF specifies that the alternate fast scanner  rep-
              resentation  (described  above  under  the --FF flag)
              should be used.

              --CCmm  directs  _f_l_e_x  to  construct  _m_e_t_a_-_e_q_u_i_v_a_l_e_n_c_e
              _c_l_a_s_s_e_s_,  which are sets of equivalence classes (or
              characters, if equivalence classes  are  not  being
              used)  that  are  commonly  used  together.   Meta-



Version 2.3                26 May 1990                         23





FLEX(1)                                                   FLEX(1)


              equivalence classes are often a big win when  using
              compressed tables, but they have a moderate perfor-
              mance impact (one or two "if" tests and  one  array
              look-up per character scanned).

              A  lone --CC specifies that the scanner tables should
              be compressed but neither equivalence  classes  nor
              meta-equivalence classes should be used.

              The  options  --CCff  or --CCFF and --CCmm do not make sense
              together  -  there  is  no  opportunity  for  meta-
              equivalence  classes if the table is not being com-
              pressed.   Otherwise  the  options  may  be  freely
              mixed.

              The  default  setting is --CCeemm,, which specifies that
              _f_l_e_x should generate equivalence classes and  meta-
              equivalence  classes.   This  setting  provides the
              highest degree of table compression.  You can trade
              off faster-executing scanners at the cost of larger
              tables with the following generally being true:

                  slowest & smallest
                        -Cem
                        -Cm
                        -Ce
                        -C
                        -C{f,F}e
                        -C{f,F}
                  fastest & largest

              Note that scanners with  the  smallest  tables  are
              usually  generated  and  compiled  the quickest, so
              during development you will usually want to use the
              default, maximal compression.

              --CCffee  is  often a good compromise between speed and
              size for production scanners.

              --CC options are not cumulative; whenever the flag is
              encountered,  the  previous -C settings are forgot-
              ten.

       --SSsskkeelleettoonn__ffiillee
              overrides the default skeleton file from which _f_l_e_x
              constructs  its  scanners.   You'll never need this
              option unless you are  doing  _f_l_e_x  maintenance  or
              development.

PPEERRFFOORRMMAANNCCEE CCOONNSSIIDDEERRAATTIIOONNSS
       The  main  design  goal  of _f_l_e_x is that it generate high-
       performance scanners.  It has been optimized  for  dealing
       well  with large sets of rules.  Aside from the effects of
       table compression on scanner speed outlined  above,  there



Version 2.3                26 May 1990                         24





FLEX(1)                                                   FLEX(1)


       are a number of options/actions which degrade performance.
       These are, from most expensive to least:

           REJECT

           pattern sets that require backtracking
           arbitrary trailing context

           '^' beginning-of-line operator
           yymore()

       with the first three all being  quite  expensive  and  the
       last two being quite cheap.

       RREEJJEECCTT  should be avoided at all costs when performance is
       important.  It is a particularly expensive option.

       Getting rid of backtracking is messy and often may  be  an
       enormous  amount  of  work  for a complicated scanner.  In
       principal, one begins by using the --bb flag to  generate  a
       _l_e_x_._b_a_c_k_t_r_a_c_k file.  For example, on the input

           %%
           foo        return TOK_KEYWORD;
           foobar     return TOK_KEYWORD;

       the file looks like:

           State #6 is non-accepting -
            associated rule line numbers:
                  2       3
            out-transitions: [ o ]
            jam-transitions: EOF [ \001-n  p-\177 ]

           State #8 is non-accepting -
            associated rule line numbers:
                  3
            out-transitions: [ a ]
            jam-transitions: EOF [ \001-`  b-\177 ]

           State #9 is non-accepting -
            associated rule line numbers:
                  3
            out-transitions: [ r ]
            jam-transitions: EOF [ \001-q  s-\177 ]

           Compressed tables always backtrack.

       The  first  few lines tell us that there's a scanner state
       in which it can make a transition on an 'o' but not on any
       other  character,  and  that  in  that state the currently
       scanned text does not match any rule.   The  state  occurs
       when  trying  to match the rules found at lines 2 and 3 in
       the input file.  If the scanner is in that state and  then



Version 2.3                26 May 1990                         25





FLEX(1)                                                   FLEX(1)


       reads  something  other than an 'o', it will have to back-
       track to find a rule which is  matched.   With  a  bit  of
       headscratching  one  can  see  that this must be the state
       it's in when it has seen "fo".  When this has happened, if
       anything  other than another 'o' is seen, the scanner will
       have to back up to simply match the 'f'  (by  the  default
       rule).

       The comment regarding State #8 indicates there's a problem
       when "foob" has been scanned.  Indeed,  on  any  character
       other  than  a  'b',  the  scanner will have to back up to
       accept "foo".  Similarly, the comment for  State  #9  con-
       cerns when "fooba" has been scanned.

       The  final  comment reminds us that there's no point going
       to all the trouble of removing backtracking from the rules
       unless  we're using --ff or --FF,, since there's no performance
       gain doing so with compressed scanners.

       The way to remove  the  backtracking  is  to  add  "error"
       rules:

           %%
           foo         return TOK_KEYWORD;
           foobar      return TOK_KEYWORD;

           fooba       |
           foob        |
           fo          {
                       /* false alarm, not really a keyword */
                       return TOK_ID;
                       }


       Eliminating backtracking among a list of keywords can also
       be done using a "catch-all" rule:

           %%
           foo         return TOK_KEYWORD;
           foobar      return TOK_KEYWORD;

           [a-z]+      return TOK_ID;

       This is usually the best solution when appropriate.

       Backtracking messages tend to cascade.  With a complicated
       set  of  rules  it's  not uncommon to get hundreds of mes-
       sages.  If one can decipher them, though,  it  often  only
       takes  a  dozen  or so rules to eliminate the backtracking
       (though it's easy to make a mistake and have an error rule
       accidentally  match a valid token.  A possible future _f_l_e_x
       feature will be to automatically add  rules  to  eliminate
       backtracking).




Version 2.3                26 May 1990                         26





FLEX(1)                                                   FLEX(1)


       _V_a_r_i_a_b_l_e  trailing  context  (where  both  the leading and
       trailing parts do not have a fixed length) entails  almost
       the  same  performance loss as _R_E_J_E_C_T (i.e., substantial).
       So when possible a rule like:

           %%
           mouse|rat/(cat|dog)   run();

       is better written:

           %%
           mouse/cat|dog         run();
           rat/cat|dog           run();

       or as

           %%
           mouse|rat/cat         run();
           mouse|rat/dog         run();

       Note that here the special '|' action does _n_o_t provide any
       savings,  and  can  even  make  things  worse (see BBUUGGSS in
       flex(1)).

       Another area where the user can increase a scanner's  per-
       formance  (and one that's easier to implement) arises from
       the fact that the longer the tokens  matched,  the  faster
       the  scanner  will  run.  This is because with long tokens
       the processing of most input characters takes place in the
       (short) inner scanning loop, and does not often have to go
       through the additional work of  setting  up  the  scanning
       environment  (e.g.,  yyyytteexxtt))  for  the action.  Recall the
       scanner for C comments:

           %x comment
           %%
                   int line_num = 1;

           "/*"         BEGIN(comment);

           <comment>[^*\n]*
           <comment>"*"+[^*/\n]*
           <comment>\n             ++line_num;
           <comment>"*"+"/"        BEGIN(INITIAL);

       This could be sped up by writing it as:

           %x comment
           %%
                   int line_num = 1;

           "/*"         BEGIN(comment);

           <comment>[^*\n]*



Version 2.3                26 May 1990                         27





FLEX(1)                                                   FLEX(1)


           <comment>[^*\n]*\n      ++line_num;
           <comment>"*"+[^*/\n]*
           <comment>"*"+[^*/\n]*\n ++line_num;
           <comment>"*"+"/"        BEGIN(INITIAL);

       Now instead of each newline requiring  the  processing  of
       another  action, recognizing the newlines is "distributed"
       over the other rules to keep the matched text as  long  as
       possible.   Note  that _a_d_d_i_n_g rules does _n_o_t slow down the
       scanner!  The speed of the scanner is independent  of  the
       number of rules or (modulo the considerations given at the
       beginning of this section) how complicated the  rules  are
       with regard to operators such as '*' and '|'.

       A final example in speeding up a scanner: suppose you want
       to scan through a file  containing  identifiers  and  key-
       words,  one  per line and with no other extraneous charac-
       ters, and recognize all the  keywords.   A  natural  first
       approach is:

           %%
           asm      |
           auto     |
           break    |
           ... etc ...
           volatile |
           while    /* it's a keyword */

           .|\n     /* it's not a keyword */

       To  eliminate  the  back-tracking,  introduce  a catch-all
       rule:

           %%
           asm      |
           auto     |
           break    |
           ... etc ...
           volatile |
           while    /* it's a keyword */

           [a-z]+   |
           .|\n     /* it's not a keyword */

       Now, if it's guaranteed that there's exactly one word  per
       line,  then we can reduce the total number of matches by a
       half by merging in the recognition of newlines  with  that
       of the other tokens:

           %%
           asm\n    |
           auto\n   |
           break\n  |
           ... etc ...



Version 2.3                26 May 1990                         28





FLEX(1)                                                   FLEX(1)


           volatile\n |
           while\n  /* it's a keyword */

           [a-z]+\n |
           .|\n     /* it's not a keyword */

       One  has  to  be careful here, as we have now reintroduced
       backtracking into the scanner.  In  particular,  while  _w_e
       know  that there will never be any characters in the input
       stream other than letters or newlines, _f_l_e_x  can't  figure
       this out, and it will plan for possibly needing backtrack-
       ing when it has scanned a token like "auto" and  then  the
       next character is something other than a newline or a let-
       ter.  Previously it would then just match the "auto"  rule
       and  be  done,  but  now  it  has  no  "auto" rule, only a
       "auto\n" rule.  To eliminate the possibility of backtrack-
       ing, we could either duplicate all rules but without final
       newlines, or, since we never expect to encounter  such  an
       input  and  therefore  don't  how  it's classified, we can
       introduce one more catch-all rule, this one which  doesn't
       include a newline:

           %%
           asm\n    |
           auto\n   |
           break\n  |
           ... etc ...
           volatile\n |
           while\n  /* it's a keyword */

           [a-z]+\n |
           [a-z]+   |
           .|\n     /* it's not a keyword */

       Compiled  with --CCff,, this is about as fast as one can get a
       _f_l_e_x scanner to go for this particular problem.

       A final note: _f_l_e_x is slow when matching  NUL's,  particu-
       larly  when a token contains multiple NUL's.  It's best to
       write rules which match _s_h_o_r_t  amounts  of  text  if  it's
       anticipated that the text will often include NUL's.

IINNCCOOMMPPAATTIIBBIILLIITTIIEESS WWIITTHH LLEEXX AANNDD PPOOSSIIXX
       _f_l_e_x  is a rewrite of the Unix _l_e_x tool (the two implemen-
       tations do not share any code, though), with  some  exten-
       sions  and incompatibilities, both of which are of concern
       to those who wish to write scanners acceptable  to  either
       implementation.   At  present, the POSIX _l_e_x draft is very
       close to the original _l_e_x implementation, so some of these
       incompatibilities  are  also  in  conflict  with the POSIX
       draft.  But the intent is that except as noted below, _f_l_e_x
       as it presently stands will ultimately be POSIX conformant
       (i.e., that those areas of conflict with the  POSIX  draft
       will  be  resolved  in _f_l_e_x_'_s favor).  Please bear in mind



Version 2.3                26 May 1990                         29





FLEX(1)                                                   FLEX(1)


       that all the comments which follow are with regard to  the
       POSIX  _d_r_a_f_t  standard  of  Summer 1989, and not the final
       document (or subsequent drafts); they are included so _f_l_e_x
       users can be aware of the standardization issues and those
       areas where _f_l_e_x may in the near  future  undergo  changes
       incompatible with its current definition.

       _f_l_e_x  is  fully  compatible  with  _l_e_x  with the following
       exceptions:

       -      The  undocumented  _l_e_x  scanner  internal  variable
              yyyylliinneennoo is not supported.  It is difficult to sup-
              port this option  efficiently,  since  it  requires
              examining  every  character scanned and reexamining
              the characters when the scanner backs  up.   Things
              get more complicated when the end of buffer or file
              is reached or a NUL is scanned (since the scan must
              then  be  restarted  with  the  proper  line number
              count), or the user uses the yyless(), unput(),  or
              REJECT  actions, or the multiple input buffer func-
              tions.

              The fix is to add rules which, upon seeing  a  new-
              line,  increment yylineno.  This is usually an easy
              process, though it can be a drag  if  some  of  the
              patterns  can  match  multiple  newlines along with
              other characters.

              yylineno is not part of the POSIX draft.

       -      The iinnppuutt(()) routine is not redefinable,  though  it
              may be called to read characters following whatever
              has been matched by a rule.  If iinnppuutt(())  encounters
              an  end-of-file  the  normal yyyywwrraapp(()) processing is
              done.   A  ``real''  end-of-file  is  returned   by
              iinnppuutt(()) as _E_O_F_.

              Input  is  instead  controlled  by  redefining  the
              YYYY__IINNPPUUTT macro.

              The _f_l_e_x restriction that iinnppuutt(()) cannot  be  rede-
              fined  is  in  accordance with the POSIX draft, but
              YYYY__IINNPPUUTT has not yet been accepted into  the  draft
              (and  probably  won't; it looks like the draft will
              simply not specify any way of controlling the scan-
              ner's input other than by making an initial assign-
              ment to _y_y_i_n_)_.

       -      _f_l_e_x scanners do not use stdio for input.   Because
              of  this,  when  writing an interactive scanner one
              must explicitly call fflush() on the stream associ-
              ated  with the terminal after writing out a prompt.
              With _l_e_x  such  writes  are  automatically  flushed
              since  _l_e_x  scanners use ggeettcchhaarr(()) for their input.



Version 2.3                26 May 1990                         30





FLEX(1)                                                   FLEX(1)


              Also, when writing interactive scanners with  _f_l_e_x_,
              the --II flag must be used.

       -      _f_l_e_x scanners are not as reentrant as _l_e_x scanners.
              In particular, if you have an  interactive  scanner
              and  an  interrupt  handler which long-jumps out of
              the scanner, and the scanner is subsequently called
              again, you may get the following message:

                  fatal flex scanner internal error--end of buffer missed

              To reenter the scanner, first use

                  yyrestart( yyin );


       -      oouuttppuutt(())  is  not  supported.  Output from the EECCHHOO
              macro is done to the  file-pointer  _y_y_o_u_t  (default
              _s_t_d_o_u_t_)_.

              The  POSIX  draft mentions that an oouuttppuutt(()) routine
              exists but currently gives no details as to what it
              does.

       -      _l_e_x  does  not  support  exclusive start conditions
              (%x), though they are in the current POSIX draft.

       -      When definitions are expanded, _f_l_e_x  encloses  them
              in parentheses.  With lex, the following:

                  NAME    [A-Z][A-Z0-9]*
                  %%
                  foo{NAME}?      printf( "Found it\n" );
                  %%

              will  not  match  the string "foo" because when the
              macro is expanded the rule is equivalent to "foo[A-
              Z][A-Z0-9]*?"   and the precedence is such that the
              '?' is associated with "[A-Z0-9]*".  With _f_l_e_x_, the
              rule will be expanded to "foo([A-Z][A-Z0-9]*)?" and
              so the string "foo" will match.  Note that  because
              of  this,  the  ^^,, $$,, <<ss>>,, //,, and <<<<EEOOFF>>>> operators
              cannot be used in a _f_l_e_x definition.

              The POSIX  draft  interpretation  is  the  same  as
              _f_l_e_x_'_s_.

       -      To specify a character class which matches anything
              but a left bracket (']'), in _l_e_x one can use "[^]]"
              but  with  _f_l_e_x  one  must use "[^\]]".  The latter
              works with _l_e_x_, too.

       -      The _l_e_x %%rr (generate a Ratfor  scanner)  option  is
              not  supported.  It is not part of the POSIX draft.



Version 2.3                26 May 1990                         31





FLEX(1)                                                   FLEX(1)


       -      If you are providing your own yywrap() routine, you
              must  include  a "#undef yywrap" in the definitions
              section (section 1).  Note that the  "#undef"  will
              have to be enclosed in %{}'s.

              The  POSIX draft specifies that yywrap() is a func-
              tion and this is very unlikely to change;  so  _f_l_e_x
              _u_s_e_r_s  _a_r_e  _w_a_r_n_e_d  that  yyyywwrraapp(())  is likely to be
              changed to a function in the near future.

       -      After a call to  uunnppuutt(()),,  _y_y_t_e_x_t  and  _y_y_l_e_n_g  are
              undefined until the next token is matched.  This is
              not the case with _l_e_x or the present POSIX draft.

       -      The precedence of the {{}} (numeric  range)  operator
              is  different.  _l_e_x interprets "abc{1,3}" as "match
              one, two, or three occurrences of  'abc'",  whereas
              _f_l_e_x  interprets it as "match 'ab' followed by one,
              two, or three occurrences of 'c'".  The  latter  is
              in agreement with the current POSIX draft.

       -      The precedence of the ^^ operator is different.  _l_e_x
              interprets "^foo|bar" as "match either 'foo' at the
              beginning  of  a  line, or 'bar' anywhere", whereas
              _f_l_e_x interprets it as "match either 'foo' or  'bar'
              if they come at the beginning of a line".  The lat-
              ter is in agreement with the current POSIX draft.

       -      To refer to yytext outside of  the  scanner  source
              file,  the  correct definition with _f_l_e_x is "extern
              char *yytext" rather than "extern  char  yytext[]".
              This  is  contrary to the current POSIX draft but a
              point on which _f_l_e_x will not be  changing,  as  the
              array  representation entails a serious performance
              penalty.  It is hoped that the POSIX draft will  be
              emended  to support the _f_l_e_x variety of declaration
              (as this is a fairly painless change to require  of
              _l_e_x users).

       -      _y_y_i_n  is  _i_n_i_t_i_a_l_i_z_e_d  by _l_e_x to be _s_t_d_i_n_; _f_l_e_x_, on
              the other hand, initializes _y_y_i_n to NULL  and  then
              _a_s_s_i_g_n_s  it  to _s_t_d_i_n the first time the scanner is
              called,  providing  _y_y_i_n  has  not   already   been
              assigned  to  a  non-NULL value.  The difference is
              subtle, but the net effect is that with _f_l_e_x  scan-
              ners,  _y_y_i_n  does  not have a valid value until the
              scanner has been called.

       -      The special table-size declarations such as %%aa sup-
              ported  by  _l_e_x  are not required by _f_l_e_x scanners;
              _f_l_e_x ignores them.

       -      The name FLEX_SCANNER is #define'd so scanners  may
              be written for use with either _f_l_e_x or _l_e_x_.



Version 2.3                26 May 1990                         32





FLEX(1)                                                   FLEX(1)


       The following _f_l_e_x features are not included in _l_e_x or the
       POSIX draft standard:

           yyterminate()
           <<EOF>>
           YY_DECL
           #line directives
           %{}'s around actions
           yyrestart()
           comments beginning with '#' (deprecated)
           multiple actions on a line

       This last feature refers to the fact that  with  _f_l_e_x  you
       can  put multiple actions on the same line, separated with
       semi-colons, while with _l_e_x_, the following

           foo    handle_foo(); ++num_foos_seen;

       is (rather surprisingly) truncated to

           foo    handle_foo();

       _f_l_e_x does not truncate the action.  Actions that  are  not
       enclosed in braces are simply terminated at the end of the
       line.

DDIIAAGGNNOOSSTTIICCSS
       _r_e_j_e_c_t___u_s_e_d___b_u_t___n_o_t___d_e_t_e_c_t_e_d         _u_n_d_e_f_i_n_e_d          or
       _y_y_m_o_r_e___u_s_e_d___b_u_t___n_o_t___d_e_t_e_c_t_e_d  _u_n_d_e_f_i_n_e_d _- These errors can
       occur at compile time.  They  indicate  that  the  scanner
       uses RREEJJEECCTT or yyyymmoorree(()) but that _f_l_e_x failed to notice the
       fact, meaning that _f_l_e_x scanned  the  first  two  sections
       looking  for  occurrences  of  these actions and failed to
       find any, but somehow you snuck some in  (via  a  #include
       file,  for  example).   Make  an explicit reference to the
       action in your _f_l_e_x input  file.   (Note  that  previously
       _f_l_e_x  supported a %%uusseedd//%%uunnuusseedd mechanism for dealing with
       this problem; this feature is still supported but now dep-
       recated,  and  will  go  away soon unless the author hears
       from people who can argue compellingly that they need it.)

       _f_l_e_x  _s_c_a_n_n_e_r  _j_a_m_m_e_d  _-  a  scanner  compiled with --ss has
       encountered an input string which wasn't matched by any of
       its rules.

       _f_l_e_x  _i_n_p_u_t  _b_u_f_f_e_r  _o_v_e_r_f_l_o_w_e_d _- a scanner rule matched a
       string long enough  to  overflow  the  scanner's  internal
       input  buffer  (16K  bytes  by  default  -  controlled  by
       YYYY__BBUUFF__SSIIZZEE in "flex.skel".  Note that  to  redefine  this
       macro, you must first ##uunnddeeffiinnee it).

       _s_c_a_n_n_e_r  _r_e_q_u_i_r_e_s  _-_8  _f_l_a_g  _-  Your scanner specification
       includes recognizing 8-bit  characters  and  you  did  not
       specify  the -8 flag (and your site has not installed flex



Version 2.3                26 May 1990                         33





FLEX(1)                                                   FLEX(1)


       with -8 as the default).

       _f_a_t_a_l _f_l_e_x _s_c_a_n_n_e_r _i_n_t_e_r_n_a_l _e_r_r_o_r_-_-_e_n_d _o_f _b_u_f_f_e_r _m_i_s_s_e_d  _-
       This  can  occur  in an scanner which is reentered after a
       long-jump has jumped out (or over) the  scanner's  activa-
       tion frame.  Before reentering the scanner, use:

           yyrestart( yyin );


       _t_o_o  _m_a_n_y  _%_t  _c_l_a_s_s_e_s_!  _- You managed to put every single
       character into its own %t class.  _f_l_e_x  requires  that  at
       least one of the classes share characters.

DDEEFFIICCIIEENNCCIIEESS // BBUUGGSS
       See flex(1).

SSEEEE AALLSSOO
       flex(1), lex(1), yacc(1), sed(1), awk(1).

       M.  E. Lesk and E. Schmidt, _L_E_X _- _L_e_x_i_c_a_l _A_n_a_l_y_z_e_r _G_e_n_e_r_a_-
       _t_o_r

AAUUTTHHOORR
       Vern Paxson, with the help of many ideas and much inspira-
       tion   from   Van   Jacobson.   Original  version  by  Jef
       Poskanzer.  The fast table  representation  is  a  partial
       implementation  of  a  design  done  by Van Jacobson.  The
       implementation was done by Kevin Gong and Vern Paxson.

       Thanks to the many  _f_l_e_x  beta-testers,  feedbackers,  and
       contributors,  especially  Casey  Leedom,  benson@odi.com,
       Keith Bostic,  Frederic  Brehm,  Nick  Christopher,  Jason
       Coughlin,  Scott  David  Daniels, Leo Eskin, Chris Faylor,
       Eric Goldman, Eric Hughes,  Jeffrey  R.  Jones,  Kevin  B.
       Kenny, Ronald Lamprecht, Greg Lee, Craig Leres, Mohamed el
       Lozy,  Jim  Meyering,  Marc  Nozell,  Esmond   Pitt,   Jef
       Poskanzer,  Jim  Roskind,  Dave Tallman, Frank Whaley, Ken
       Yap, and those whose names have slipped my marginal  mail-
       archiving  skills  but whose contributions are appreciated
       all the same.

       Thanks to Keith Bostic, John  Gilmore,  Craig  Leres,  Bob
       Mulcahy,  Rich  Salz,  and  Richard Stallman for help with
       various distribution headaches.

       Thanks to Esmond Pitt and Earle Horton for 8-bit character
       support;  to  Benson Margulies and Fred Burke for C++ sup-
       port; to Ove Ewerlid for the basics of support for  NUL's;
       and  to Eric Hughes for the basics of support for multiple
       buffers.

       Work is being done on extending _f_l_e_x to generate  scanners
       in  which  the  state machine is directly represented in C



Version 2.3                26 May 1990                         34





FLEX(1)                                                   FLEX(1)


       code rather than tables.  These scanners may well be  sub-
       stantially faster than those generated using -f or -F.  If
       you are working in this area and are interested in compar-
       ing  notes  and  seeing  whether  redundant  work  can  be
       avoided, contact Ove Ewerlid (ewerlid@mizar.DoCS.UU.SE).

       This work was primarily done when I was at the  Real  Time
       Systems  Group  at  the  Lawrence  Berkeley  Laboratory in
       Berkeley, CA.  Many thanks to all there for the support  I
       received.

       Send comments to:

            Vern Paxson
            Computer Science Department
            4126 Upson Hall
            Cornell University
            Ithaca, NY 14853-7501

            vern@cs.cornell.edu
            decvax!cornell!vern




































Version 2.3                26 May 1990                         35


