
               Link Grammar Parser - Patched Version
               -------------------------------------
                       Version 4.4.2

This directory contains a *patched* version of the final original
release of the Link Grammar Parser.  It has been patched to fix a
few bugs, add a few enhancements, and, in general, make the Link
Grammar Parser easier to use. This version includes Java bindings.

It is released under the BSD license (which is GPL compatible),
making it freely available for private and commercial use, with
few restrictions.  The terms of the license are given in the
LICENSE file included with this software, and also available at
http://www.link.cs.cmu.edu/link/license.html.

Please see the web page http://www.abisource.com/projects/link-grammar/
for more information.  The original website, including documentation,
is located at http://www.link.cs.cmu.edu/link

CONTENTS of this directory:

   LICENSE                  The license describing terms of use

   link-grammar/*.c         The program.  (Written in ANSI-C)

   data/en/4.0.dict         The file containing the dictionary definitions.

   data/en/4.0.knowledge    The post-processing knowledge file.

   data/en/4.0.constituents The constituent knowledge file.

   data/en/4.0.affix        The affix file.

   data/en/tiny.dict        A small sample dictionary.

   data/en/words/*          A directory full of word lists.

   data/en/4.0.batch        This is a batch file of sentences (both grammatical
                            and ungrammatical ones) that are handled correctly
                            by this release of the system.  Feed this into the
                            parser with "./link-parser < 4.0.batch"

   configure                The GNU configuration script

   msvc6                    Microsoft Visual-C project files

UNPACKING and signature verification:

   The system is distributed using the normal tar.gz format; it can be
   extracted using the "tar -zxf link-grammar.tar.gz" command at the
   command line.

   The files have been digitally signed to make sure that there was no
   corruption of the dataset during download, and to help ensure that
   no malicious changes were made to the code internals by third
   parties. The signatures can be checked with the gpg command:

   gpg --verify link-grammar-4.3.10.tar.gz.asc

   which should generate output identical to (except for the date):

   gpg: Signature made Tue Feb 26 19:41:07 2008 CST using RSA key ID C0389241
   gpg: Good signature from "Linas Vepstas (current active email; others are inactive) <linasvepstas@gmail.com>"

   Alternately, the md5 checksums can be verified. These do not provide
   cryptographic security, but they can detect simple corruption. To
   verify the checksums, issue "md5sum -c MD5SUM" at the command line.

CREATING the system:

   To compile the link-grammar shared library and demonstration program,
   at the command line, type:

        ./configure
        make

   To install, change user to "root" and say

        make install

   This will install the lib-link-grammar.so library into /usr/local/lib
   the header files in /usr/local/include/link-grammar
   and the dictionaries into /usr/local/share/link-grammar.

   The /usr/local install target can be over-ridden using the
   standard GNU configure --prefix option, so for example:

      ./configure --prefix=/opt/link-grammar

   By using pkg-config (see below), non-standard install locations
   can be automatically detected.

   The system has been tested and works well on 32 and 64-bit Linux
   systems, as well as on many Microsoft Windows systems, under
   various different Windows development environments.

BUILDING on Windows:

    The easiest way to have link-grammar working on MS Windows, especially
    if you want the Java bindings, is to use MinGW/MSYS, a minimalist
    GNU environment for Windows that includes gcc, bash, automake, and many
    other utilities.  Download and install MinGW, MSYS and MSYS-DTK from
    http://mingw.org .

    Then build and install link-grammar with

       ./configure
       make
       make install

    If you used the standard installation paths, the directory /usr/ is mapped
    to C:\msys\1.0, so after 'make install', the libraries and executable
    will be found at C:\msys\1.0\local\bin and the dictionary files at
    C:\msys\1.0\local\share\link-grammar.

    In order to use the Java bindings you'll need to build two extra DLLs, running
    from the link-grammar base directory the following commands:

       cd link-grammar

       gcc -g -shared -Wall -D_JNI_IMPLEMENTATION_ -Wl,--kill-at \
       .libs/analyze-linkage.o .libs/and.o .libs/api.o .libs/build-disjuncts.o .libs/command-line.o \
       .libs/constituents.o .libs/count.o .libs/error.o .libs/extract-links.o .libs/fast-match.o \
       .libs/idiom.o .libs/massage.o .libs/post-process.o .libs/pp_knowledge.o .libs/pp_lexer.o \
       .libs/pp_linkset.o .libs/prefix.o .libs/preparation.o .libs/print-util.o .libs/print.o \
       .libs/prune.o .libs/read-dict.o .libs/resources.o .libs/string-set.o .libs/tokenize.o \
       .libs/utilities.o .libs/word-file.o .libs/word-utils.o \
       -o /usr/local/bin/link-grammar.dll

       gcc -g -shared -Wall -D_JNI_IMPLEMENTATION_ -Wl,--kill-at \
       .libs/jni-client.o /usr/local/bin/link-grammar.dll \
       -o /usr/local/bin/link-grammar-java.dll

    This will create link-grammar.dll and link-grammar-java.dll in the directory
    c:\msys\1.0\local\bin . These files, together with link-grammar-*.jar, will
    be used by Java programs.

    Alternately, Windows users who have access to the "Microsoft Visual C"
    product may wish to use the project files in the msvc6 directory.

RUNNING the program:

   To run the program issue the Unix command:

       ./link-parser

   This starts the program.  Help is available there with "!help".  A
   number of user-settable variables control what happens.  "!var" shows
   these variables and their current values.

   The dictionaries contain some utf-8 punctuation. These may generate
   errors for users in a non-utf-8 locale, such as the "C" locale.
   The locale can be set, for example, by saying

       export LANG=en_US.UTF-8

   at the shell prompt.

   The program can run in batch mode for testing the system on a large
   number of sentences.  The following command runs the parser on
   a file called 4.0.batch

       ./link-parser < 4.0.batch

   The line "!batch" near the top of 4.0.batch turns on batch mode.  In
   this mode sentences labeled with an initial "*" should be rejected
   and those not starting with a "*" should be accepted.  The current
   batch file does report some errors; there are a collection of
   failing parses located at the end. Work is ongoing to fix these.

USING the parser in your own applications:

   There is a API (application program interface) to the parser.  This
   makes it easy to incorporate it into your own applications.  The API
   documented on the web site.

USING pkg-config:

   To make compiling and linking easier, the current release uses
   the pkg-config system. To determine the location of the link-grammar
   header files, say `pkg-config --cflags link-grammar`  To obtain
   the location of the libraries, say `pkg-config --libs link-grammar`
   Thus, for example, a typical makefile might include the targets:

      .c.o:
         cc -O2 -g -Wall -c $< `pkg-config --cflags link-grammar`

      $(EXE): $(OBJS)
         cc -g -o $@ $^ `pkg-config --libs link-grammar`

JAVA bindings:

   This release includes Java bindings.  Thier use is optional. 

   The bindings will be built automatically if jni.h can be found.
   Some common java JVM distributions (most notably, the ones from Sun)
   place this file in unusual locations, where it cannot be
   automatically found.  To remedy this, make sure that JAVA_HOME is
   set. If jni.h still cannot be found, specify the location with the
   CPPFLAGS variable: so, for example,

      export CPPFLAGS="-I/opt/jdk1.5/include/:/opt/jdk1.5/include/linux"
   or
      export CPPFLAGS="-I/c/java/jdk1.6.0/include/ -I/c/java/jdk1.6.0/include/win32/"

   Please note that the use of /opt is non-standard, and most system
   tools will fail to find packages installed there.

MULTI-THREADED USE:

   The link-grammar library should be thread-safe as of version 4.3.6,
   although this has not yet been tested very well, and may possibly be
   buggy.  To enable the thread-safe version of the library, run
   ./configure like so:

      ./configure --enable-pthreads

   The resulting library is thread-safe for all parsing and general
   use, with the exception of the startup/initialization code.  That
   is, the initial open/creation of the parser dictionary should be
   done only once, from a single thread, and parsing should not be
   started until after the dictionary has been read and initialized.

   Parse options can be set on a per-thread basis, with the exception
   of verbosity, which is treated as a global.

   The library uses pthreads for the few spots where it needs to manage
   per-thread storage. It is possibly quite reasonable to abolish these 
   few spots, thus making the library not only thread-safe, but thread
   agnostic (i.e. not require pthreads).

   The following exceptions and special notes apply:

      error.c -- uses pthreads to access error message area.
      utilities.c -- has global "verbosity". Uses pthreads for tracking
                     memory usage. The memory usage code is just about
                     obsolete, and could probably be discarded.
      jni-client.c - uses per-thread struct, but needs to be attached
                     to JNIEnv somehow.
      malloc-dbg.c - not thread safe, not normally used;
                     only for debugging.
      prefix.c - not thread-safe, but doesn't need to be; used only
                 during initialization, and only if binreloc turned
                 on.
      pp_lexer.c -- autogened code, original lex sources lost.
                    This is only used when reading dictionaries,
                    during initialization, and so doesn't need
                    to be thread safe.
      utilities.h -- Windows doesn't have a thread-safe random-number
                     generator, which means link-grammar for Windows
                     is not currently thread-safe.


COMMERCIAL use:

   The original authors would be pleased to discuss alternate
   licensing terms and/or additional research work, with those
   wanting to use this system under a paid contract.

   Academic products or papers should reference the various
   link grammar publications.

ADDRESSES

   If you have any questions, or find any bugs, please feel free
   to send a note to the mailing list:

      link-grammar@googlegroups.com

   Although all messages should go to the mailing list, the current
   maintainers can be contacted at:

     Dom Lachowicz - <domlachowicz@gmail.com>
     Linas Vepstas - <linasvepstas@gmail.com>

   The original authors of the Link Grammar parser are:

     Daniel Sleator                    sleator@cs.cmu.edu
     Computer Science Department       412-268-7563
     Carnegie Mellon University        www.cs.cmu.edu/~sleator
     Pittsburgh, PA 15213

     Davy Temperley                    dtemp@theory.esm.rochester.edu
     Eastman School of Music           716-274-1557
     26 Gibbs St.                      www.link.cs.cmu.edu/temperley
     Rochester, NY 14604

     John Lafferty                     lafferty@cs.cmu.edu
     Computer Science Department       412-268-6791
     Carnegie Mellon University        www.cs.cmu.edu/~lafferty
     Pittsburgh, PA 15213


TODO -- Working Notes:
   Some working notes.
   Quotations: Currently, tokenize.c ignores all ASCII double-quotes 
      (grep for "quote_found" in the source). However, it does not do this
      for the various "curly" UTF8 quotes, such as ‘these’ and “these”.
      This results is some ugly parsing for sentences containing such 
      quotes. (Note that these are in 4.0.affix).

   tokenize.c: build_sentence_expressions() contains some
      English-language-specific morphology guessing.  This should be
      refactored out, as it doesn't really apply to non-English
      languages.

   no-links-cross: Link Grammar uses a constraint that all linakges must
      be planar graphs. While this is mostly correct, it would probably
      be more correct to use "landmark transitivity" as articulated by
      Hudson in the Word Grammar theory.  This might allow the elimination
      of most or all post-processing rules.

      This is done by making each link directional: one end of the link
      is the parent. (e.g. noun, noun-modifier: the noun is the parent).
      Parents are landmarks for children.  Transitivity is applied to
      parent-child relationships. Specifically, the no-links-cross
      rule is replaced by two landmark transitivity rules:

      -- If B is a landmark for C, then  A is also a type-L landmark for C
      -- If A is a landmark for C, then B is also a landmark for C

      where type-L means either a right-going or left-going link.

     See http://goertzel.org/ProwlGrammar.pdf for details.

  "to be fishing": Link grammar offers four parses of "I was fishing for
     evidence", two of which are given low scores, and two are given
     high scores. Of the two with high scores, one parse is clearly bad.
     Its links "to be fishing.noun" as opposed to the correct
     "to be fishing.gerund". That is, I can be happy, healthy and wise,
     but I certainly cannot be fishing.noun.  This is perhaps not
     just a bug in the structure of the dictionary, but is perhaps
     deeper: link-grammar has little or no concept of lexical units
     (i.e. collocations, idioms, institutional phrases), which thus
     allows parses with bad word-senses to sneak in.

     The goal is to introduce more knowledge of lexical units into LG.

     Different word senses can have different grammar rules (and thus,
     the links employed reveal the sense of the word): for example:
     "I tend to agree" vs. "I tend to the sheep" -- these employ two
     different meanings for the verb "tend", and the grammatical
     constructions allowed for one meaning are not the same as those
     allowed for the other. Yet, the link rules for "tend.v" have
     to accomodate both senses, thus making the rules rather complex.
     Worse, it potentially allows for non-sense constructions.
     If, instead, we allowed the dictionary to contain different
     rules for "tend.meaning1" and "tend.meaning2", the rules would
     simplify (at the cost of inflating the size of the dictionary).

  "holes" in collocations:
     The link-grammar does not support circumpositions (or even more 
     complicated multi-word structures), when such structures have 
     "holes" in the middle. For example, the adposition:

        ... from [xxx] on.

            "He never said another word from then on."
            "I promise to be quiet from now on."
            "Keep going straight from that point on."
            "We went straight from here on."
            ... from there on.
            "We went straight, from the house on to the woods."
            "We drove straight, from the hill onwards."
 
    Note that multiple words can fit in the slot [xxx].
    Note the tangling of another prepositional phrase: 
    "... from [xxx] on to [yyy]"

    More complicated collocations with holes include 
    "First.. next...", and "if ... then ..."


  incremental parsing: to avoid a combinatorial explosion of parses,
    it would be nice to have an incremental parsing, phrase by phrase,
    using a Viterbi-like algorithm to obtain the parse. Thus, for example,
    the parse of the last half of a long, run-on sentence should not be
    sensitive to the parse of the begining of the sentence.

    Doing so would help with combinatorial explosion. So, for example,
    if the first half of a sentence has 4 plausible parses, and the
    last half has 4 more, then link-grammar reports 16 parses total.
    It would be much, much more useful to instead be given the
    factored results: i.e. the for plausbile parses for the
    first half, and the four plausible parses for the last half.
    The lower combinatoric stress would ease the burden on
    downstream users of link-grammar.

    (This somewhat resembles the application of construction grammar
    ideas to the link-grammar dictionary).

