/** @file mainpage.c
 *
 * Contains the content for the main page of the documentation.
 * Also has a number of pages that don't really fit anywhere else.
 *
 * There's no point in compiling this file since it only contains
 * documentation but there's no harm either.
 */




/** @mainpage

libre2c tries to make it much easier use re2c scanners 
sacrificing re2c's amazing flexibility.  This
documentation is for re2c version VERSION.


\section examples Examples

The best way to learn about libre2c is to read some examples.
These examples are arranged in order from simplest to strangest.

- \ref numscancoupled -- A very simple scanner and parser
	contained in a single file.

- \ref numscanmodular -- Continues the previous example.
	It separates numscan-coupled into a separate
    scanner and parser.  Also shows how to read data using fopen/fread.

- \ref startstates -- A scanner that demonstrates switching between
	start states while looking for C and C++ comments.

- \ref cmpscan -- A utility that compares two data sources byte-for-byte.
	This example shows that libre2c can be useful even if you
	don't use re2c to generate a scanner.

There is also the skeleton project in the examples directory.
It's a project where you can just
fill in the blanks to create an re2c scanner.

\section reference Reference

- \ref writing_parsers

- \ref writing_scanners

- \ref writing_readprocs

\subsection readproc_builtins Builtin Readprocs

A ::readproc feeds data to the scanner.  Here are some routines that
libre2c provides to use the built-in readprocs.
If none of these fits your needs, see \ref writing_readprocs.

- readfd_attach() -- attaches an already-open file descriptor to an already-created scanner.
- readfd_open() -- creates a scanner and sets it up to read from a filepath.
- readfp_attach() -- attaches an already-open FILE* to an already-created scanner.
- readfp_open() -- creates a scanner and sets it up to read from a filepath.
- readmem_attach() -- causes a scanner to read directly from a memory block.
- readmem_init_str() -- initializes a scanner to read from a C string.
- readrand_attach() -- feeds the scanner a pseudorandom byte stream.

*/





/** @page writing_parsers How to Use a Scanner

This page tells you how to set up your code to call an re2c scanner.
See \ref writing_scanners to learn how to write re2c scanners.
See \ref numscancoupled and \ref numscanmodular for a tutorial.

Using a re2c scanner is quite easy:

* decide how you'll read the data.  You'll use a readproc for this.
  See either \ref readproc_builtins or \ref writing_readprocs).

* decide how you'll buffer the scanner data.  Will you malloc a buffer?
  buffer?  Will you create it on the stack?  Are you using a readproc
  that doesn't even require a buffer (such as readmem.c)?

Then:

* Create the buffer if needed.
* Create the scanstate, point it at the buffer.
* Attach the readproc.
* Attach the scanner.

Now you're ready to scan.

* Call scan_next_token() until it returns either 0 (eof) or negative (error).

See the above two tutorials for working code examples.

 */





/** @page writing_scanners Writing Scanners

This page tells you how to write a libre2c scanner.
See \ref writing_parsers to learn how to call these scanners.
See \ref numscancoupled and \ref numscanmodular for a tutorial.

Scanners consist of code generated by re2c, surrounded by
a tiny bit of boilerplate support code to ensure tokens are
properly maintained.

Scanners typically consist of two files:

- A header file defining the tokens and the routine(s) to attach
the scanner to the readproc.

- A source file containing the re2c source and a small amount
of bolierplate.

You can use scanner.h and scanner.re as a starting point for your scanners.

 */



/** @page writing_readprocs Writing Readprocs

The readproc is the routine responsible for filling the re2c
scanner's buffer with data.

libre2c includes some readprocs that read from most common
data sources (libc FILE*, Unix file descriptors, a memory block, etc).
See \ref readproc_builtins for a list of readprocs included with
libre2c.

If a builtin readproc does everything you need, then there is no need
to read this page further.  If you need to write a custom readproc,
however, this page should tell you everything you need.
See \ref writing_scanners to learn how to write an re2c scanner.

\section readproc_arguments Arguments

\section readproc_retval Return Value

See \ref error_handling for more information on how libre2c handles errors.

\section readproc_shortreads Short Reads

re2c uses a pull model, where the scanner calls the readproc whenever
it needs more data.  re2c has recently been modified to support a
push model as well but libre2c does not support this yet.
This means that it is not easy to parse data from any source that
may return short reads (such as a pipe, socket, fifo, Unix device,
etc).  Your readproc should never return a short read so you'll
need to use buffering if the underlying data source does.

Until the push mode was added, re2c didn't not handle any data
source that would return short reads well.  libre2c has
not (yet) been updated to take advantage of push mode.

How does the scanner tell the difference between EOF and EAGAIN?  If re2c
notices that it might run out of data before scanning a token, it
calls the readproc once to ensure that there is as much data in the
buffer as possible.  This ensures that you can always scan a token
exactly as big as the buffer.

You only need to know this if you're writing your own read functions.

Your readproc must first make room in the buffer for more data.
The read_shiftbuffer() routine is the best way to do this.
Once you have freed up room at the end of the buffer, fill it
with data.

@returns
- 0 when there's no more data (EOF).
- positive: the number of bytes read (note that re2c does not work
well with short reads).
- negative: an error value.  The value returned depends on the readproc.
Many readprocs return Unix error codes such as -EWOULDBLOCK.



All charptrs in the scanstate structure are declared const to help
ensure that scanner or parser writers don't
accidentally end up modifying the buffer as it is being scanned.
This means that your read routine must cast them to be mutable
(char*) before reading them.  It is a libre2c convetion
that only the readproc may modify a scanner's buffer.

The caller assumes that the read routine will always fill the buffer
up as much as possible.  Therefore, if the buffer isn't entirely full,
then it knows that the EOF is probably at the end of the data.
There is currently no way in re2c for a token to span a short read.
This is fine for files but not so good for pipes, network
sockets, anything that is packetized or works in realtime.

The shift technique was chosen over a ringbuffer because we should rarely
have to shift data.  If you find that your file has gigantic tokens
and you're burning a lot of cpu shifting partial tokens from the end
of the buffer to the start, you might be tempted to use a ring buffer
instead of a shift buffer.  Alas, re2c itself can't handle ringbuffers or
tokens split across buffer boundaries (and neither can most scanners
that I'm aware of).

 */





/** @page linenos Line Numbers

Because it costs CPU cycles and many scanners don't need it,
neither re2c scanners nor libre2c keep track of line numbers.

It is trivial, however, to modify your scanner to keep track of line
numbers itself.  And, if it won't cause a significant performance
drop, it's highly suggested.  All you do is ensure that your scanner
matches \\n as a single token and increment scanstate::line every time
it is matched.

A scanner's attach routine normally leaves scanstate::line set to 0 if
it doesn't properly handle line numbers, or sets it to 1 if it does.

Here is a scanner that doesn't properly keep track of line numbers.
It groups numbers and non-numbers into contiguous tokens.  Newlines
are included along with every other non-digit character in the STRING
token.  This means that STRING tokens can span multiple lines.

<pre>
	[0-9]+    { return NUMBER;  }
	[^0-9]+   { return STRING; }
</pre>

To fix this, we first set the line number in the attach proc.

<pre>
scanstate* numscan_attach(scanstate *ss)
{
	ss->state = numscan_start;
	scan_set_line(ss, 1);
}
</pre>

Then, in the re2c scanner, we specify that "\\n" is a token and increase the
line number whenever this token is matched:

<pre>
	[0-9]+     { return NUMBER;  }
	[^0-9\\n]+ { return STRING; }
	[\\n]      { scan_inc_line(ss); return NEWLINE }
</pre>

The \ref numscanmodular example shows a full 
scanner that properly keeps track of line numbers.

*/




/** @page error_handling Error Handling

libre2c is meticulous at checking errors so you don't have to be.

If an error occurs while scanning, it is stored within the scanner.
From now on, any time you try to pull a token, the error code will
be returned.  This means that you can defer all error checking until
you're finished scanning.
In addition to making your code smaller and more readable, this
allows you to handle all your error

<pre>
	i1 = scan_next_token(ss);
	// An error may have occurred but we don't need worry about it here.
	i2 = scan_next_token(ss);
	// If i1 returned an error, we're guaranteed that i2 contains the same error.
	i3 = scan_next_token(ss);
	if(i3 > 0) {
		// We know that i1 and i2 were valid too.
	} else if(i3 == 0) {
		// We know that i3 was not returned because we hit EOF.
		// i1 and i2 may or may not be 0 as well.
	} else { // i3 < 0
		// There was a read error during parsing!
	}
</pre>

\section errvals Error Values

There are only two types of errors that may occur during scanning.
The readproc may encounter OS-level trouble, such as an I/O error, and
the scanner may encounter a text sequence that sends it into an invalid
state.  Both of these errors are reported using the return value from
scan_next_token():

- Positive return values are tokens or scanner errors.
- Negative return values are ::readproc errors (the exact meaning of the negative value depends on the readproc).
- 0 indicates a normal EOF.

If a token is negative, it represents an error reported by a readproc.
Scanners always return positive tokens, even if the token represents
an error state.
This ensures that the scanner and
readproc error spaces will never clash, and that the scanner will not
cause the readproc's error handling to enter into a weird state.

\subsection readerrvals Readproc Errors

When the readproc reports an error, the ::scanstate enters an error
state that can only be reset by scanstate_reset.  Readproc errors
cause the scanning process to halt permanently.

Readproc errors are always negative numbers.

\subsection scanerrvals Scanner Errors

The scanner decides how it will handle error reporting.  Some scanners
may find it useful to latch into an error state when an error token
is returned.  Others may return the error token, then try to resume
scanning as further tokens are fetched.

Scanner errors are always positive numbers defined by the scanner
as errors.  Read the particular scanner's documentation to see how
it is handled.

*/
