FCODE suite: Difference between revisions

Latest revision as of 01:35, 16 January 2016

What is the OpenBIOS FCODE Suite?

OpenBIOS provides a sophisticated set of FCODE utilities:

the tokenizer toke
the detokenizer detok
and a PCI rom header utility.
a portable implementation of forth local values

These files are offered without any warranty. If you experience problems, please contact the OpenBIOS mailinglist.

Downloading the OpenBIOS FCode Suite

The latest version of the OpenBIOS FCode Suite is 1.0.2 (released 2006-10-30)

Source Code

View the Sources online. Available as a ZIP-File: FCode-utils 1.0.2

See the ChangeLog for a list of changes since 1.0.1. A hand crafted html document describes some more changes.

Documentation

There are three documents, all in html format, plus a sub-directory of templates that provide common formatting support.

It is important that these be kept in the same directory, as there are some links from one file to another.

The documents are User's Guides to:

the New Features of the Tokenizer,
the Detokenizer, and
the Local Values feature, which is mentioned briefly in the Tokenizer User's Guide and described fully in the Local Values document.

These documents are also part of the source code repository.

There is also doxygen generated documentation available

doxygen documentation for toke 0.6.10
doxygen documentation for toke 1.0.0
TODO: doxygen documentation for toke 1.0.2

Executables for three platforms

While you can find a couple of executables here we strongly recommend that you compile the FCode toolchain from the sources above so you gain from the integration work and fixes that have been done since these executables have been created.

There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility. (Binaries)

There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.

There are two variants of each version: One that has level-2 Optimization and one that has no optimization at all, which I provided for purposes of Debugging. Optimization causes some routines and variables to become obscured and inaccessible to debuggers, and also re-arranges the sequence of execution in a way that can become confusing during single-stepping.

And finally, for each, there is a "stripped" and an "unstripped" executable image. The "unstripped" image has an extension of "unstripped"; the "stripped" image has no extension.

There are separate directories for the Debug and Optim(ized Level)2 variants.

Under each are sub-directories for the different platforms, within which the executable images reside.

All binaries are also available in a single Tar-File: Binaries.tar.bz2

Local Value Support

Includ-able tokenizer-source files for Local Values Support (explained in one of the User's Guide documents). Five files:

One supplies the basic functionality
the second adds a development-time facility
the third generates a variant behavior ("Global" scope rather than scope limited to a single Device-Node)
and the fourth combines the "Global" variant behavior with the development-time facility.
The fifth allows the choice of combinations to be governed by command-line switches, and is probably the best to use with Makefiles in commercial development and production environments.

There is commentation in each one explaining how it is to be used.

Available as part of the OpenBIOS FCODE suite.

Todo

A list of "Still To Be Done" items, excerpted from the commentation in the Sources

The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions.

This file is a collection of all of them in a single convenient location.

Unit-Test Suite

The suite of unit-test cases

This is the accumulation of test-cases that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.

Run the unit-test cases with

$ make tests

Test Tools

The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run.

The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes:

AutoExec automates the execution, and
AutoCompare automates the comparison.

There is commentation in each explaining how it is used.

Unit-Test Suite Logs

These can be used as base-lines for comparison against future versions, or, if so be, versions compiled for additional platforms.

Note that a comparison of these against each other will not yield exact identity. Some of the test-cases, for instance, code the current date and time, others display a complete file-path, and still one other attempts to load a file for encoding using a syntax that is erroneous on some O/S's but not on others.

All in all, five or six file differences will be expected to be reported by AutoCompare.

The results from a run of the Unit-Test Suite on the X86/Cygwin platform.

The results from a run of the Unit-Test Suite on the PowerPC/Linux platform.

The results from a run of the Unit-Test Suite on the PowerPC/AIX platform.

Coverage Reports

The test suite has been run using gcov/lcov to produce graphical code coverage reports.

Kudos and Thanks

The OpenBIOS FCODE Suite has been significantly enhanced to meet commercial grade requirements by David Paktor. Regarding to code readability and stability, he made the FCODE suite the best part of OpenBIOS. And it's probably the best FCODE development environment out there. Thank you, David!

@@ Line 1: / Line 1: @@
 == What is the OpenBIOS FCODE Suite? ==
-OpenBIOS provides a sophisticated set of FCODE utilities: A tokenizer, a detokenizer, and a PCI rom header utility.
+OpenBIOS provides a sophisticated set of FCODE utilities:
+* the tokenizer '''toke'''
+* the detokenizer '''detok'''
+* and a PCI rom header utility.
+* a portable implementation of forth local values
-In August 2006 David Paktor, of the IBM Corporation, sent us a comprehensive set of contributions to the OpenBIOS FCODE suite. We have not yet fully proofread and integrated these changes into the official OpenBIOS development tree, but in the meantime we want to offer you the opportunity to download the files directly.
+These files are offered without any warranty. If you experience problems, please contact the [[Mailinglist|OpenBIOS mailinglist]].
-These files are offered without any warranty. If you experience problems, please contact the OpenBIOS mailinglist.
-== Downloading the OpenBIOS FCode Suite? ==
+== Downloading the OpenBIOS FCode Suite ==
-* Modified Sources and Makefiles
+'''The latest version of the OpenBIOS FCode Suite is 1.0.2 (released 2006-10-30)'''
-  Available as a Tar-File:  TokzrNewSrcFiles.tar.bz2
-* User's Guide documents
+=== Source Code ===
-      There are three documents, all in html format, plus a sub-directory of templates that provide common formatting support.
-      It is important that these be kept in the same directory, as there are some links from one file to another.
-      The documents are User's Guides to:
-* the New Features of the Tokenizer,
-* the Detokenizer, and
-* the Local Values feature, which is mentioned briefly in the Tokenizer User's Guide and described fully in the Local Values document.
-      Available as a Tar-File:  Documents.tar.bz2
-* Executable binaries for three platforms
+View the [https://github.com/openbios/fcode-utils Sources] online. Available as a ZIP-File:  [https://github.com/openbios/fcode-utils/archive/v1.0.2.zip FCode-utils 1.0.2]
-      There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility.
-      There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.
-      There are two variants of each version: One that has level-2 Optimization and one that has no optimization at all, which I provided for purposes of Debugging. Optimization causes some routines and variables to become obscured and inaccessible to debuggers, and also re-arranges the sequence of execution in a way that can become confusing during single-stepping.
-      And finally, for each, there is a "stripped" and an "unstripped" executable image. The "unstripped" image has an extension of "unstripped"; the "stripped" image has no extension.
-      There are separate directories for the Debug and Optim(ized Level)2 variants.
-      Under each are sub-directories for the different platforms, within which the executable images reside.
-      Available as a Tar-File:  Binaries.tar.bz2
-* Includ-able tokenizer-source files for Local Values Support (explained in
+See the [https://raw.githubusercontent.com/openbios/fcode-utils/v1.0.2/ChangeLog ChangeLog] for a list of changes since 1.0.1.
-one of the User's Guide documents)
+A hand crafted html document describes [http://www.openbios.org/data/visiblediffs/V01_versus_V2x/ some more changes].
-      Five files: One supplies the basic functionality, the second adds a development-time facility, the third generates a variant behavior ("Global" scope rather than scope limited to a single Device-Node) and the fourth combines the "Global" variant behavior with the development-time facility. The fifth allows the choice of combinations to be governed by command-line switches, and is probably the best to use with Make-files in commercial development and production environments.
-      There is commentation in each one explaining how it is to be used.
-      Available as a Tar-File:  LoclValSupport.tar.bz2
-* A list of "Still To Be Done" items, excerpted from the commentation in the Sources
+=== Documentation ===
-      The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions. This file is a collection of all of them in a single convenient location.
+There are three documents, all in html format, plus a sub-directory of templates that provide common formatting support.
-      Available as a Tar-File:  StillToBeDonesList.tar.bz2
+It is important that these be kept in the same directory, as there are some links from one file to another.
+The documents are User's Guides to:
+* the [http://www.openbios.org/data/fcodesuite/Documents/toke.html New Features of the Tokenizer],
+* [http://www.openbios.org/data/fcodesuite/Documents/detok.html the Detokenizer], and
+* the [http://www.openbios.org/data/fcodesuite/Documents/localvalues.html Local Values feature], which is mentioned briefly in the Tokenizer User's Guide and described fully in the Local Values document.
+These documents are also part of the source code repository.
+There is also doxygen generated documentation available
+* [http://openbios.org/data/toke/toke-0.6.10 doxygen documentation for toke 0.6.10]
+* [http://openbios.org/data/toke/toke-1.0 doxygen documentation for toke 1.0.0]
+* TODO: doxygen documentation for toke 1.0.2
+=== Executables for three platforms ===
+While you can find a couple of executables here we strongly recommend that you compile the FCode toolchain from the sources above so you gain from the integration work and fixes that have been done since these executables have been created.
+There are three programs: the Tokenizer, the Detokenizer and the ROMHeaders utility. ([http://www.openbios.org/data/fcodesuite/Binaries Binaries])
+There is a version for each of three platforms (i.e., combinations of Processor and O/S): Cygwin running on an X86, GNU Linux running on a Power-PC, and AIX running on a Power-PC.
+There are two variants of each version: One that has level-2 Optimization and one that has no optimization at all, which I provided for purposes of Debugging. Optimization causes some routines and variables to become obscured and inaccessible to debuggers, and also re-arranges the sequence of execution in a way that can become confusing during single-stepping.
+And finally, for each, there is a "stripped" and an "unstripped" executable image. The "unstripped" image has an extension of "unstripped"; the "stripped" image has no extension.
+There are separate directories for the Debug and Optim(ized Level)2 variants.
+Under each are sub-directories for the different platforms, within which the executable images reside.
+All binaries are also available in a single Tar-File:  [http://www.openbios.org/data/fcodesuite/Binaries.tar.bz2 Binaries.tar.bz2]
+=== Local Value Support ===
+Includ-able tokenizer-source files for [https://github.com/openbios/fcode-utils/tree/master/localvalues Local Values Support] (explained in one of the User's Guide documents). Five files:
+* One supplies the [https://github.com/openbios/fcode-utils/tree/master/localvalues/LocalValuesSupport.fth  basic functionality]
+* the second adds a [https://github.com/openbios/fcode-utils/tree/master/localvalues/LocalValuesDevelSupport.fth development-time facility]
+* the third generates a variant behavior ([https://github.com/openbios/fcode-utils/tree/master/localvalues/GlobalLocalValues.fth "Global" scope] rather than scope limited to a single Device-Node)
+* and the fourth [https://github.com/openbios/fcode-utils/tree/master/localvalues/GlobalLocalValuesDevel.fth combines the "Global" variant behavior with the development-time facility].
+* The fifth [https://github.com/openbios/fcode-utils/tree/master/localvalues/TotalLocalValuesSupport.fth allows the choice of combinations to be governed by command-line switches], and is probably the best to use with Makefiles in commercial development and production environments.
+There is commentation in each one explaining how it is to be used.
+Available as part of the OpenBIOS FCODE suite.
+=== Todo ===
+A list of "Still To Be Done" items, excerpted from the commentation in the Sources
+The source files have, scattered among their commentation, an occasional item discussing a feature or implementation detail that might be worth attention in future revisions.
+This file is a [https://github.com/openbios/fcode-utils/tree/master/TODO collection of all of them] in a single convenient location.
 == Unit-Test Suite ==
-* The suite of unit-test cases
+=== The suite of unit-test cases ===
-      This is the accumulation of test-cases that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.
+This is the [https://github.com/openbios/fcode-utils/tree/master/testsuite accumulation of test-cases] that were created in the course of development. Some of these are a straightforward invocation of a feature, others are convoluted combinations of features whose interaction needed to be carefully watched, and still others are collections of coding errors, for purposes of verifying the Error-detection capabilities of the Tokenizer. They are grouped into sub-directories representing broad categories.
-      One small set of items might be of particular interest: In a subdirectory called TokMisc/OldTkzr are two test-case source files, called ContErr0.fth and SimplStrEscs.fth together with output files that have .OldTkzr. in their names. These are the "Log" file, FCode binary output, and DeTokenization of the FCode binary, respectively, as they appear after having been run through the "old" tokenizer (version 0.6.10, dated 2005-10-15).
-      The proper operation of the Test-suite has some dependency on the directory structure. Specifically, it expects the LocalValues directory, containing the Local Values Support files, to be a sibling of the parent (an "uncle" if you will) of the directory in which the Test-suite resides. There are sym-links that will clarify this.
-      Available as a Tar-File:   TestSrc.tar.bz2
-* The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run
+Run the unit-test cases with
-      The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes: AutoExec automates the execution, and AutoCompare automates the comparison. Again, there is commentation in each explaining how it is used.
-      Available as a Tar-File:  TestTools.tar.bz2
+ $ make tests
+=== Test Tools ===
+The tools to run the Unit-Test Suite as a batch and examine the results for changes relative to the results from a previous run.
+The process of manually running a unit-test and comparing against the previous output, after every change, became unwieldy, especially when it came to running the entire suite of tests. These scripts were developed to automate both processes:
+* [https://github.com/openbios/fcode-utils/tree/master/testsuite/AutoExec AutoExec] automates the execution, and
+* [https://github.com/openbios/fcode-utils/tree/master/testsuite/AutoCompare AutoCompare] automates the comparison.
+There is commentation in each explaining how it is used.
 == Unit-Test Suite Logs ==
-These can be used as base-lines for comparison against future versions, or, if so be, versions compiled for additional platforms. Note that a comparison of these against each other will not yield exact identity. Some of the test-cases, for instance, code the current date and time, others display a complete file-path, and still one other attempts to load a file for encoding using a syntax that is erroneous on some O/S's but not on others.
+These can be used as base-lines for comparison against future versions, or, if so be, versions compiled for additional platforms.
+Note that a comparison of these against each other will not yield exact identity. Some of the test-cases, for instance, code the current date and time, others display a complete file-path, and still one other attempts to load a file for encoding using a syntax that is erroneous on some O/S's but not on others.
 All in all, five or six file differences will be expected to be reported by AutoCompare.
-* The results from a run of the Unit-Test Suite on the X86/Cygwin platform
+* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-x86-cygwin X86/Cygwin] platform.
-      Available as a Tar-File:  X86_Cygwin_TestLog.tar.bz2
+* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-ppc-linux PowerPC/Linux] platform.
+* The results from a run of the Unit-Test Suite on the [https://github.com/openbios/fcode-utils/tree/master/testlogs/testlogs-ppc-aix PowerPC/AIX] platform.
+== Coverage Reports ==
+The test suite has been run using gcov/lcov to produce graphical code coverage reports.
+* [http://www.openbios.org/data/coverage-fcode-suite-1.0.2/ coverage report for FCODE suite 1.0.2]
+* [http://www.openbios.org/data/coverage-toke-1.0.2/ coverage report for toke 1.0.2]
+* [http://openbios.org/data/toke/coverage/ coverage report for toke 1.0.0]
+== Kudos and Thanks ==
-* The results from a run of the Unit-Test Suite on the Power-PC/Linux platform
+<div style="margin-top:0.5em; margin-bottom:0.5em; padding:0.5em 0.5em 0.5em 0.5em; background-color:#efefff; align:right; border:1px solid #aabbcc;">
-      Available as a Tar-File: PPC_GNU_Linux_TestLog.tar.bz2
+The OpenBIOS FCODE Suite has been significantly enhanced to meet commercial grade requirements by [mailto:David@paktor.biz David Paktor]. Regarding to code readability and stability, he made the FCODE suite the best part of OpenBIOS. And it's probably the best FCODE development environment out there. '''Thank you, David!'''
+</div>
-* The results from a run of the Unit-Test Suite on the Power-PC/AIX platform
-      Available as a Tar-File:  PPC_AIX_TestLog.tar.bz2
+__NOEDITSECTION__