David Korn
2009-06-04 18:09:03 UTC
Subject: comments and information about nmake and related AST tools
--------
Here is some high level information that I hope helps you to understand
nmake.
The aim of nmake is to provide a way to build software with minimal
specification that works with many languages on many systems.
There are many facets of nmake that work together to provide a nearly
complete solution. I will discuss some of these here.
First of all, nmake is designed to look a lot like make in that it
consists of assignments, assertions, and rules. However, unlike make,
it tries to separate out what the user needs to supply from what the
system needs to know to build software. It does this by providing a
scripting language where much of the intelligence resides.
nmake rules are collected in this order: the standard base rules
(loaded by default), global rules (loaded by --global=file), implicit
makefile includes for :foo: assertions (loaded from
../lib/make/foo.mk), explicit makefile includes (loaded by "include
file"), explicit makefiles (loaded by --file=file overriding the
default), and finally the implicit makefile (the first of Nmakefile,
nmakefile, Makefile, makefile.) Any rules file can be compiled to
speed up runtime execution. By default nmake compiles the user
makefile into Makefile.mo and automatically recompiles it when the
makefile or an of its include prerequisites change. The rules handle
most common configuration management operations.
The :PACKAGE: assertion, as mentioned earlier, tries to find the
include and library directories for a given package, and appends these
to the header and library options when compiling. The tgz and pax
rules generate tarballs of all source files (implicit and explicit)
referenced in the makefile. Users can add files to the tarball with an
assertion
:: file_list
Action blocks (rules) can be written in either shell or nmake. Most
rules in user makefiles are written as shell scripts, whereas most
rules in the files in ../lib/make are written as nmake scripts. Unlike
make, nmake treats the complete rule as a single action rather than
running a shell for each line. In fact nmake runs the shell as a
coprocess sending actions to it and getting back results. It can run
actions in parallel and even run them on multiple machines using another
tool named coshell.
nmake recognizes that a single piece of software may be divided into
multiple directories and allows assertions that specify what
directories and order to search for files of different types. This
information is then used to generate default -I and -L compiler
options.
nmake also eliminates a lot of user specification by handling implicit
dependencies. Unlike make, which runs an auxiliary command to get
dependencies, nmake scans each source and generated file for
dependencies based on file type. The rules for scanning files of a
given type are defined in nmake scan rules. Scan rule specifications
allow each language to have a syntax for comments, quoting, includes,
ifdefs, and defines so that new languages can be added. For example,
when we worked on a cobol project, we were able to define the rules for
cobol source.
When nmake generates implicit information it stores the results into a
statefile in the current directory. The statefile contains the
prerequisites and timestamps for include files and the values for
defines that were specified as dependencies in the makefile. This
saves the need to regenerate the implicit information for files that
have not changed since the last make. The statefile also contains the
timestamp of each file. nmake bases "out of date" on timestamp
changes, whereas make relies on prerequisite timestamp newer than
target timestamp. Thus, restoring a source file to an earlier version
will trigger a recompilation with nmake but not with make.
A makefile containing the single assertion
:MAKE:
controls all makefiles in the current directory and recursively its
subdirectories. All of the makefiles are scanned for library and
command prerequisites, and nmake is run, in the correct order, in each
of the subdirectories that contain a makefile. This is how all AST
packages are organized. A single "nmake install" run from the
$INSTALLROOT/src directory build and installs all of the AST components
in the correct order; add another AST package, and next time it will
also be built in the correct order. There is no need to maintain a
manual ordering list, or to run make two or more times. (nmake
recursion considered essential.)
nmake works with a few other tools to provide portability. One of
these is probe. probe is a shell script that determines
command-specific options and provides symbolic names for option
abstractions. For example, the make probe information for the C
compiler (located in ../lib/probe/C/make, hashed compiler-specific file
name), provides symbolic names like CC.OPTIMIZE for the option to
compile with the optimizer turned on. The CC.DLL option can be used
for building a shared or dynamically linked library.
When nmake encounters the pathname for a new C compiler it runs probe
to generate the probe information corresponding to that compiler. The
probe information is generated once and shared among all nmake users.
This allows user makefiles to use symbolic option names that are
compatible with all target architectures -- all compiler specific
information is confined to the probe information files.
Another tool that nmake works with is iffe (If FEature Exists), a shell
script that builds and runs commands to answer any number of questions
about what exists and works on target systems. It is another
declarative language that allows simple questions like "does this
header exist" or "is this function found in one of the specified
libraries" and allows users to ask complex questions by inserting small
C programs or shell scripts that attempt to answer the question. The
output of iffe is a header file that can be included in any program
containing macros that can be tested and/or #includes to other headers.
The default nmake rules generate
FEATURE/foo
by running
iffe features/foo
Thus iffe is run implicitly when building software rather than having
to run iffe or configure before running nmake. Moreover, unlike
configure, iffe allows configuration files to be broken into smaller
pieces rather than generating one centralized that causes a world
re-build when it changes.
Another portability tool is mamake. Since all makefiles for the AST
software are written using nmake, there needs to be a way to bootstrap
build nmake itself. nmake has the ability to generate an intermediate
language that contains the information in a makefile in a way that can
be processed without nmake or make. This format was developed about 20
years ago and is called a MAM (Make Abstract Machine) file. If this
were to be done today, its format would likely be xml. The MAM file
can be generated by nmake, make, gmake, and other makes with sufficient
instrumentation. The mamake program builds using the description in
the generated MAM files and provides a way to build nmake.
The final tool is the package command which ties together all of the
AST tools into packages. It is a shell script that should run with all
the known UNIX shells. It generates, retrieves, reads, sets up the
viewpath environment, builds and installs (via mamake and nmake) source
and binary package tarballs.
One final note about nmake. In addition to building software, nmake and
coshell can be used as a job scheduler. We were able to emulate a commercial
scheduler that runs on the mainframe by converting the scheduling files
into nmake files and running them.
David Korn
***@research.att.com
--------
Here is some high level information that I hope helps you to understand
nmake.
The aim of nmake is to provide a way to build software with minimal
specification that works with many languages on many systems.
There are many facets of nmake that work together to provide a nearly
complete solution. I will discuss some of these here.
First of all, nmake is designed to look a lot like make in that it
consists of assignments, assertions, and rules. However, unlike make,
it tries to separate out what the user needs to supply from what the
system needs to know to build software. It does this by providing a
scripting language where much of the intelligence resides.
nmake rules are collected in this order: the standard base rules
(loaded by default), global rules (loaded by --global=file), implicit
makefile includes for :foo: assertions (loaded from
../lib/make/foo.mk), explicit makefile includes (loaded by "include
file"), explicit makefiles (loaded by --file=file overriding the
default), and finally the implicit makefile (the first of Nmakefile,
nmakefile, Makefile, makefile.) Any rules file can be compiled to
speed up runtime execution. By default nmake compiles the user
makefile into Makefile.mo and automatically recompiles it when the
makefile or an of its include prerequisites change. The rules handle
most common configuration management operations.
The :PACKAGE: assertion, as mentioned earlier, tries to find the
include and library directories for a given package, and appends these
to the header and library options when compiling. The tgz and pax
rules generate tarballs of all source files (implicit and explicit)
referenced in the makefile. Users can add files to the tarball with an
assertion
:: file_list
Action blocks (rules) can be written in either shell or nmake. Most
rules in user makefiles are written as shell scripts, whereas most
rules in the files in ../lib/make are written as nmake scripts. Unlike
make, nmake treats the complete rule as a single action rather than
running a shell for each line. In fact nmake runs the shell as a
coprocess sending actions to it and getting back results. It can run
actions in parallel and even run them on multiple machines using another
tool named coshell.
nmake recognizes that a single piece of software may be divided into
multiple directories and allows assertions that specify what
directories and order to search for files of different types. This
information is then used to generate default -I and -L compiler
options.
nmake also eliminates a lot of user specification by handling implicit
dependencies. Unlike make, which runs an auxiliary command to get
dependencies, nmake scans each source and generated file for
dependencies based on file type. The rules for scanning files of a
given type are defined in nmake scan rules. Scan rule specifications
allow each language to have a syntax for comments, quoting, includes,
ifdefs, and defines so that new languages can be added. For example,
when we worked on a cobol project, we were able to define the rules for
cobol source.
When nmake generates implicit information it stores the results into a
statefile in the current directory. The statefile contains the
prerequisites and timestamps for include files and the values for
defines that were specified as dependencies in the makefile. This
saves the need to regenerate the implicit information for files that
have not changed since the last make. The statefile also contains the
timestamp of each file. nmake bases "out of date" on timestamp
changes, whereas make relies on prerequisite timestamp newer than
target timestamp. Thus, restoring a source file to an earlier version
will trigger a recompilation with nmake but not with make.
A makefile containing the single assertion
:MAKE:
controls all makefiles in the current directory and recursively its
subdirectories. All of the makefiles are scanned for library and
command prerequisites, and nmake is run, in the correct order, in each
of the subdirectories that contain a makefile. This is how all AST
packages are organized. A single "nmake install" run from the
$INSTALLROOT/src directory build and installs all of the AST components
in the correct order; add another AST package, and next time it will
also be built in the correct order. There is no need to maintain a
manual ordering list, or to run make two or more times. (nmake
recursion considered essential.)
nmake works with a few other tools to provide portability. One of
these is probe. probe is a shell script that determines
command-specific options and provides symbolic names for option
abstractions. For example, the make probe information for the C
compiler (located in ../lib/probe/C/make, hashed compiler-specific file
name), provides symbolic names like CC.OPTIMIZE for the option to
compile with the optimizer turned on. The CC.DLL option can be used
for building a shared or dynamically linked library.
When nmake encounters the pathname for a new C compiler it runs probe
to generate the probe information corresponding to that compiler. The
probe information is generated once and shared among all nmake users.
This allows user makefiles to use symbolic option names that are
compatible with all target architectures -- all compiler specific
information is confined to the probe information files.
Another tool that nmake works with is iffe (If FEature Exists), a shell
script that builds and runs commands to answer any number of questions
about what exists and works on target systems. It is another
declarative language that allows simple questions like "does this
header exist" or "is this function found in one of the specified
libraries" and allows users to ask complex questions by inserting small
C programs or shell scripts that attempt to answer the question. The
output of iffe is a header file that can be included in any program
containing macros that can be tested and/or #includes to other headers.
The default nmake rules generate
FEATURE/foo
by running
iffe features/foo
Thus iffe is run implicitly when building software rather than having
to run iffe or configure before running nmake. Moreover, unlike
configure, iffe allows configuration files to be broken into smaller
pieces rather than generating one centralized that causes a world
re-build when it changes.
Another portability tool is mamake. Since all makefiles for the AST
software are written using nmake, there needs to be a way to bootstrap
build nmake itself. nmake has the ability to generate an intermediate
language that contains the information in a makefile in a way that can
be processed without nmake or make. This format was developed about 20
years ago and is called a MAM (Make Abstract Machine) file. If this
were to be done today, its format would likely be xml. The MAM file
can be generated by nmake, make, gmake, and other makes with sufficient
instrumentation. The mamake program builds using the description in
the generated MAM files and provides a way to build nmake.
The final tool is the package command which ties together all of the
AST tools into packages. It is a shell script that should run with all
the known UNIX shells. It generates, retrieves, reads, sets up the
viewpath environment, builds and installs (via mamake and nmake) source
and binary package tarballs.
One final note about nmake. In addition to building software, nmake and
coshell can be used as a job scheduler. We were able to emulate a commercial
scheduler that runs on the mainframe by converting the scheduling files
into nmake files and running them.
David Korn
***@research.att.com