//////////////////////////////////////////////////////////////////////////////// // $Id: releases_public.txt 549 2006-07-25 23:05:26Z jdenny $ // Authors: 2003/09/12 : Joel Edward Denny. // Purpose: Public release log for SHP. // // License: // // Copyright (c) 2003-2006 Clemson University. // // This file is part of SHP (Simplified Hypertext Preprocessor). // // SHP is free software; you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by // the Free Software Foundation; either version 2 of the License, or // (at your option) any later version. // // SHP is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with SHP; if not, write to the Free Software // Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA //////////////////////////////////////////////////////////////////////////////// ################################################################################ # shp-0.14 ################################################################################ Release Date: 2006/07/25 Posted on joeldenny.org. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- Configure: - When checking for a perl package, the required package version is now reported. - In a TECNET environment, the default prefix is now "/tecnet/shp/0.14". Now tested with: - GCC 4.1.0 on GNU/Linux. - Perl v5.8.0 on Solaris. - Perl v5.8.7 on GNU/Linux. - Perl v5.8.6-4 on Cygwin. - Python 2.4.3 on Solaris. - Python 2.4.1 on GNU/Linux. - Python 2.4.3-1 on Cygwin. - Cygwin DLL 1.5.19-4. - Automake 1.9.6. - Bison 2.3. SHP's version history is now maintained by Subversion: - CVS keywords have been replaced with SVN keywords in the header comments of all source files. - In "./README.txt", the `Distribution packaging procedure' instructions have been updated accordingly. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Some deadlocks may have been eliminated under GNU/Linux. See test suite notes below. Configure: - Previously, a Python check was broken under at least Solaris 5.9's "/bin/sh". This sometimes sent make into an infinite loop invoking the configure script. That shouldn't happen anymore. - In maintainer-mode, make is now able to rebuild "configure" when necessary. Previously, it stopped with an error about missing m4 dependencies. Build: - Flex should no longer warn that there's a rule that cannot be matched. Instead, it should warn if the default rule can be matched. That is, I'm using `flex -s' now, so I removed my catch-all rule. doc/releases_public.txt (this file): - shp.0.12 and shp.0.13 are now spelled shp-0.12 and shp-0.13 since these releases used GNU Autotools. Better organization conforming to current popular practices: - All m4 files previously in "config/" are now in "m4/". - The old "config/" is now named "build-aux/". - The test suite directory is now named "tests/" rather than "test/". Test suite: - Its directory is now named "tests/". See above. - Test case "outside/alarm_timeout_during_blocked_read.neg.script" should no longer deadlock under GNU/Linux. - For test case failures, the diffs of expected vs. actual output are now generated with `diff -u'. This is easier to read than the previous `diff -c'. README.txt: Minor cleanup for readability. ################################################################################ # shp-0.13 ################################################################################ Release Date: 2005/08/18 Internal Tag: shp-0-19 Posted on joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- WARNING: this change is not backward compatible. The talk directive now generates an error page if the process writes anything to stderr. The close-server directive now generates an error page if the process terminates with a non-zero exit status or if the process has any non-empty output pipe: - Added clean-read directive: - It checks that a process hasn't written any data to a specified output pipe. It is especially useful for halting upon any error message on a stderr pipe. - It's the same as the read directive except: - The clean-read directive generates an error page in any case that the read directive's replacement text wouldn't have been the empty string. - Thus, the replacement text for a successful clean-read directive is always the empty string. - Added clean-close directive, which is the same as the close directive except: - It cannot close individual input pipes. It only closes processes. - It generates an error page if either: - Any output pipe for the specified process contains anything. For delimited output pipes, even empty delimited blocks constitute an error. - The exit status of the specified process is non-zero. - The replacement text is always just the empty string. - Modified the talk directive to use the new clean-read directive for stderr rather than accumulate stderr in a SHP variable. - Modified the close-server directive to call only the clean-close directive. Thus: - The replacement text will now be the empty string. - Servers must close without any error and with all output pipes drained. - Modified the open-server directive not to initialize the SHP variable process_name.stderr since it is no longer used by the talk or close-server directives. WARNING: this change is not backward compatible. The open directive (and thus the open-server and open-server-with-stdin directives) now generates an error page for failure to exec a process: - Previously, the failure would not be noticed until a read of stderr or a close directive. - Even then, it wasn't a SHP error (except with the new clean-read, talk, and close-server directives). Building and packaging: - SHP's distributed pre-built parser source for use when maintainer mode is disabled is now generated by bison 2.0. - Configure with maintainer mode enabled now requires bison 2.*. - If maintainer mode is enabled, configure now stops with an error message if the correct version of flex or bison isn't found. - Configure now checks the perl version using perl's own `use 5.8.0;'. That is, it now depends on perl to determine if the found version of perl is compatible with the desired version. Thus, the check is more lenient. The tutorial: - Previously stated that I had not ever tested SHP under Cygwin with a web server, but I've now tested with Apache under Cygwin. The reference manual: - Added note that web servers like Apache capture a CGI's stderr to the server error log. - Updated note about a future command-line flag to specify error page templates: it will likely be a directive instead. Moreover, SHP will likely process the templates rather than just copy them verbatim. The todo list: - Replaced todo item 2004/03/26-new-2 (about specifying output language with a command-line flag) and 2003/10/11-fix-2 (about escaping process names in error pages) with 2005/08/15-fix-1 (about adding a set-error-template directive). - Added todo item 2005/08/15-internal-1 about eliminating precedence declarations from parser. - Added `Informal summary of other plans' section to so that users can see what I'm thinking about. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Fixed a number of problems related to pipes to processes: - Previously, if SHP's stdin was the terminal, a process connected to SHP's stdin became a suspended background process and thus wasn't able to actually read SHP's stdin: - Now, instead of letting the process inherit SHP's stdin as the process's specified input pipe, SHP creates a thread to copy data from its stdin to the process's input pipe. - Thus, the process no longer becomes a suspended background process, and it can read SHP's stdin. - Because of the previous fix, there is now a problem on non-POSIX systems like Cygwin where read() is not a cancellation point: - SHP may deadlock even past an alarm if (1) you've specified that one of its processes should connect to SHP's stdin and (2) SHP's stdin has not closed. - This usually happens when SHP's stdin is the terminal and you haven't closed SHP's stdin by pressing Ctrl-D: - In this case, just pressing Enter in that terminal window is often sufficient to terminate SHP once the deadlock occurs. - On non-POSIX systems like Cygwin where write() is not a cancellation point, SHP previously would deadlock until alarm when trying to close a process that had a full input pipe but that wouldn't read that pipe: - This was most likely to occur when the process failed to start or when it died unexpectedly. - SHP would wait indefinitely for the writer thread to join, but the writer thread was blocked in write() since the input pipe was full. - In order to prevent this, SHP now drains the process's end of the pipe before trying to join the thread. - Previously, I intended that getc() be *the* cancellation point for output pipe reader threads, but it isn't actually a cancellation point: - Thus, until a process's reader threads returned because the process closed its output pipes, SHP would wait indefinitely for the threads to join. - That means that if a process ignored a SIGKILL and did not close its output pipes, SHP could deadlock. - To work around this potential problem, output pipe reader threads no longer use FILE* for buffering data. Instead, they use read() and implement their own buffers of size BUFSIZ. - On non-POSIX systems like Cygwin where read() is not a cancellation point, the deadlock may still be possible. - Note that I'm not sure that a process *can* actually ignore SIGKILL, so this may not be a real problem. - Whether workarounds for non-cancellable read() and write() are employed is based on pthread_cancel() feature tests rather than Cygwin environment detection. - Previously, under Cygwin, deadlock could occur if an alarm happened to activate when SHP was in a pthread_cond_wait() waiting for data on a queue from a reader thread for a process's output pipe: - The problem was that the thread's output queue mutex may have been left locked. It seems that this only happened under Cygwin. - When SHP tried to join this reader thread (which couldn't be cancelled since read() is not a cancellation point under Cygwin), that thread might have tried to enqueue data and thus blocked waiting for the mutex lock. The result was deadlock. - I was able to observe this problem consistently at one point while developing SHP 0.13 when the code was in such a state that the reader thread always had something to enqueue. - The obvious solution was to unlock the mutex before trying to join the thread. I didn't make this fix Cygwin-specific because it doesn't hurt on other platforms and who knows what other platforms might require it as well. Previously, there were a few (rare?) process handling error conditions for which SHP did not recognize that a process needed to be killed upon SHP termination: - The most notable was probably a waitpid() failure due to receipt of a signal. (Note that this is different from a waitpid() success indicating that the child received a signal.) - I never observed this problem in practice, but it was apparent in the code. Process termination and waitpid() issues: - In the case of child process termination due to signal (WIFSIGNALED()): - The SHP error message previously stated `Process terminated with error code: '. It now correctly says `signal' instead of `error code'. - SHP would erroneously write a message to stderr about a failed kill() if there were no other processes in the process group. It no longer does. - Removed todo item 2004/06/15-fix-1 about whether some errors reported by waitpid() perhaps shouldn't generate an error page: - I still feel that termination with non-zero exit status may be intended by the user in some cases (for example, `grep'). - However, termination with signal and other forms of termination errors are usually unintended. If the user doesn't like this, he can use perl or sh to invoke and trap process termination errors. - Removed todo item 2004/06/15-fix-2 about whether an error reported by waitpid() would lead to an error from kill() in Process__delete() if there were no other processes in the process group: - For termination due to signal, the answer was yes, and I fixed this as described above. - I don't believe the problem will arise for the other error cases. Previously forgot to hide SIGALRM while deleting the run directive's process: - That is, if an alarm were to activate at exactly the right moment, the data structure for the run directive process might have been left in an inconsistent state. - Thus, when the alarm tried to kill and delete all processes, the effect of trying to delete the run directive process would have been undefined. One possible effect was a seg fault. - I never actually observed this happen for certain, but it could have. Error and usage reporting: - In the case of a failure to kill a process group, the previous error message on stderr was hard to understand in web server error logs: - SHP now writes a better error message that mentions SHP and has a timestamp. - Location information is now at the top of error pages since some error messages are much longer now. It's more logical anyway since the flow is towards the specific location of the error and then the error message is given. - In error pages from alarm executables and the run directive: - Moved exit status before stderr so that it is easier to find when stderr is long. - Delimited start and end of stderr with a line. - In the case of an exit status failure with no stderr: - Previously, they printed a lot of whitespace in the place of the contents of stderr. - They now print `none' instead. - The new behvaior matches the behavior of the new clean-close and close-server directives. - Placed quotes around process name reported. - An alarm process is now named "user-specified alarm" rather than "Alarm". - Quoted the filename in the `Unable to open' error message. - Slightly improved the explanation of `--' in the usage message. - The WIFSIGNALED() error message fix listed above. Building and packaging: - Previously, the "bootstrap" script wasn't being distributed, so I've added it to `dist_noinst_SCRIPTS'. - Previously, `./configure --help' did not display the default prefix correctly. It does now. - Previously, in the dedicated installation directory error message from configure, the list of corrections to be made was always empty. The list now prints as intended. - "README.txt" is now built and listed in CLEANFILES only for maintainer mode since (1) the user need not build it since it is distributed and (2) it doesn't make sense to list it in DISTCLEANFILES, but, if it's a built file, `make distcheck' complains if it's not in DISTCLEANFILES. - Previously, due to my ignorance of VPATH, building SHP would fail in the "templates/" subdirectory if the source and build directories were separate and the relative paths between the two were not the same. This shouldn't be a problem anymore. - Previously, in order to make strsignal() available in , "Process.c" contained `#define _GNU_SOURCE' if the platform was GNU/Linux: - This followed the Linux man page for strsignal(). - However, at some point it became necessary to have `#define _GNU_SOURCE 1' instead, and so compilation on GNU/Linux began generating warnings about strsignal() being undefined again. - From now on, #define'ing _GNU_SOURCE will be handled by autoconf's AC_GNU_SOURCE. - Moreover, configure now checks for strsignal() in case it's not available at all. If not, a replacement function is provided that simply returns "[strsignal() unavailable]". - Test suite "Makefile.am.generate" script problems: - Every "Makefile.am.generate" script in the test suite now removes "Makefile.am.tmp" before trying to create a new one. Previously, if one was still sitting around, it would append to it and possibly create a bad "Makefile.am". - Previously, "Makefile.am.generate" generated ` \' lines in "Makefile.am" if "Makefile.in" was present but not if it wasn't. Thus, "bootstrap" on a completely clean directory generated a different "Makefile.am" than subsequent invocations of "Makefile.am.generate". This caused unnecessary rebuilding. Changed the sed commands so that these ` \' lines are no longer generated. - The source and build directory no longer have to be the same for `make distcheck'. - Any missing library, header, or function is now reported during configure rather than during make. This is more convenient for the user. - Added configure checks for , , and . I'm not sure what my rationale for not having them before was. Perhaps I thought they were so standard there was no need to check... but the check doesn't hurt. - Previously, "doc/user_guide/Makefile.am" set execute permission for all HTML files. It doesn't anymore. The test suite: - Previously, some python server test cases failed for some versions of python that don't print the filename with compile() errors. Added a configure feature test for python so that the test output can be adjusted accordingly. - In "run_tests.pl": - Previously, a warning about updating the test case output source was issued in the case when no test case output was actually updated due to exit status failures. Now, the warning is only issued if at least one test case output file update was actually attempted. Moreover, if any update failure was detected, a separate warning is issued for that. - `-h' is no longer accepted as an abbreviation of `--help'. First, this is more consistent with the SHP executable itself. Second, I believe that flags like `--help' and `--version' that essentially act as stand-alone subcommands shouldn't be possible to bundle. Added copyright notice to several files in "doc/" that were missing it. In "README.txt": - First header is now `Copyright and license', which is more accurate and consistent than just `Copyright'. - Corrected typo: `--exec_prefix' --> `--exec-prefix'. - `Compatibility notes for the general user' stated that the `yyunput' warning comes from flex. It comes from gcc for flex-generated code. - The example "run_tests.pl" command now starts with "./". - Under `Distribution packaging procedure': - Now explains that `cvs tag' and `cvs export' are necessary. - Forgot to mention running "./bootstrap". The reference manual and tutorial: - Example name for symbolic link to SHP executable is now shp-VERSION rather than shp.VERSION. This is more consistent with the tar filename chosen by autotools. The reference manual: - Previously in the iterate-namespace examples for a perl hash, I forgot to escape `\' with the replace directive. - Added note to the read directive documentation stating explicitly that any delimiter char is not included in the replacement text although it is removed from the pipe. - In the explanation of the python_server, reworded the previously ambiguous definition of a multiline code block. - Cleaned up description of the setenv directive some: - Removed some obviously unnecessary uses of passive voice. - Added important note that the point of executable invocation is the *close* of an open, open-server, or run directive. That is, a setenv directive within the open directive will affect it, but this is probably bad style. - Corrected a couple of typos in character entities that made the W3C validator generate warnings for this page. - There are 6 process directives that affect the process table not 4. - Various other minor corrections. The tutorial: - Corrected the examples by removing the blank lines appearing before the XML declarations. Added note about this requirement from the XML standard. - s/blank line/empty line/g in many places for accuracy. - Reorganized `The Talk Directive' to show the examples before giving the excruciating details. - Moved the example of variables going out of scope from `The Perl and Python Error Messages' section to `The Talk Directive' section where it seems more appropriate. - Previously, I used the words `rare' and `most' to describe when web user permissions are determined by group or user instead of other. I've removed the word `rare' and changed `most' to `many' since I really don't know the frequency. - I don't want `code block' to become the standard SHP terminology. It's a `talk directive'. Thus, I tried to replace uses of `perl code block' and `python code block' with `perl talk directive' and `python talk directive' except where I need to draw a distinction between the directive and the code in its end parameter. - Other various rewordings for clarity. The todo list: - Removed todo item 2003/09/29-fix-2 about the browser stop button's effect on SHP and the trapping of signals in order to kill processes. I'm not sure it made any sense. #------------------------------------------------------------------------------- # Reorganization. #------------------------------------------------------------------------------- WARNING: this change is not backward compatible. Removed `--pipe_output_through' command-line flag: - To achieve the same effect, surround the contents of your SHP file with the more flexible run directive, which was first available in release 0.12. - For now, using `--pipe_output_through' will generate an error page stating that the run directive should be used instead. WARNING: this change is not backward compatible. The empty string namespace is now always global: - That is, the reprocess directive always automatically adds it to the list of shared namespaces. - Thus, specifying it as a shared namespace in the parameter list of the reprocess directive has no effect. - The reprocess directive has always syntactically required that at least one shared namespace be specified. Thus, it is now possible to write a reprocess directive that doesn't share any namespace besides what must always be shared: <$reprocess><$:> Text to reprocess. <$/reprocess> WARNING: this change is not backward compatible. Variable names in the empty string namespace are now always read-only: - This impacts any directive that modifies variable values: - The extract or increment directive will generate an error page if the specified variable name is in the empty string namespace. - The assign directive will generate an error page if the specified variable name or destination namespace is in the empty string namespace. - The default directive: - Will generate an error page if both (1) the specified variable name is in the empty string namespace and (2) a default value is specified. The rationale is that, regardless of the whether the default value actually ever gets used, it doesn't make sense to specify a default value for a variable name if you would never be allowed to assign it. - Will *not* necessarily generate an error page when a default value is *not* specified. In this case, the default directive behaves the same for all variable names: it will generate an error page only if the variable name is not currently assigned. The rationale is that, regardless of whether you yourself are allowed to assign a variable name, it *does* make sense to declare that it must somehow currently have a value. - Some future release of SHP will automatically assign special values to some variable names in the empty string namespace. WARNING: this change is not backward compatible. Process names in the empty string namespace are now illegal. This affects all process directives that manipulate the process table: - Open, close, and clean-close. - Read, clean-read, and write. Reformated this release log for consistency with the BOOF release log, which looked nicer. ################################################################################ # shp-0.12 ################################################################################ Release Date: 2005/01/10 Internal Tag: shp-0-17 Posted on joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- SHP now uses GNU autotools as its build system. This should greatly improve portability and ease of installation. Note the following related changes: - The install procedure is simplified dramatically. You should read "README.txt" for the details, but the procedure is basically the familiar one: ./configure make make check make install - "src/" now has two subdirectories, both of which contain source to be compiled and installed to "bin/": - "servers/", which contains the source for the server scripts. - "shp/", which contains the source for the SHP executable. - Renamed the "test_suite/" subdirectory to "test/", which is apparently more standard. - SHP now recognizes the environment variable 'SHP_INSTALL_DIRECTORY': - If set, it will override SHP's compiled default install directory. Currently, this affects: - Where SHP looks for error page templates. - Where it looks for the server scripts. - The replacement text for the SHP-install-directory directive. - This is mainly useful for testing SHP in its build directory before installing it. - "test/run_tests.pl": - It now has exit status 1 if any test fails. It has exit status 0 if all tests pass. Thus, `make check', which calls it, will report its status correctly. - It now sets the SHP_INSTALL_DIRECTORY env variable to the build directory before running SHP so that `make check' can call it before installation. - It now clobbers "tmp_stdout.txt" and "tmp_stderr.txt" rather than stopping with an error message that they still exist... because that was just really annoying with `make check'. - It now copies all test results to "test.log" and informs the user of this. Thus, `make check' can complete without interaction and the user can still read the test results. - In reference manual: - Added a brief new section under `Basic Concepts' called `Setup'. It references the "README.txt" for detailed installation instructions, and it describes the new SHP_INSTALL_DIRECTORY env var. - Removed all references to configurable parameters in "src/Makefile" since they're now configurable via flags to the configure script instead. "README.txt" and `configure --help' document those flags. Other reference manual additions: - Improved the namespace explanation and terminology to make it more intelligible in light of XML's usage of namespaces. - Added notes about the future possibility of formal argument declarations for input streams. Added variable increment directive `<$inc>': - It accepts one parameter, a variable name. - If the value of the specified variable is not an unsigned decimal integer, it generates an error page. Otherwise, it increments the value. - This is very nice for creating lists that can later be copied to an embedded language for manipulating. That is, it can be used to increment the size of the list so that the previous and future number of elements are computed at run-time rather than hard-coded. Added run directive: - The run directive is the easiest, cleanest, and least flexible of the process directives. It is useful when you wish to invoke an executable, wait for it to terminate, and capture its stdout, and you expect no failures of any kind. Specifically, if (1) any error occurs while opening or closing the executable, (2) the executable writes anything to stderr, or (3) it returns a non-zero exit status, SHP will generate an error page. - I created the run directive mainly as a more flexible version of the --pipe_output_through flag. For example, a template can use the run directive rather than depending on the CGI script to specify a required --pipe_output_through process. The old version of the forward-input-whitespace-collapse directive, '<$->', was deprecated as of SHP release 0.7. Now, it generates an error page that says that `<$"->' should be used instead. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Accelerated reader threads significantly in the case of large (delimited) nodes on an output pipe: - The problem was a call on pthread_setcancelstate() in the reader thread code. Previously, a reader thread enabled and disabled thread cancellation around every single char it read. Now, it enables and disables it around each entire node instead. Obviously, this greatly reduces the number of calls on pthread_setcancelstate() and significantly boosts efficiency. - In a test case that read 15003080 bytes into one node from an output pipe, this shaved off about 7 secs. The test case was a web page, which included a great deal of other computation. The total time for the page dropped from about 15 to 8 secs after this fix. I didn't want to take the time to hack out all of the other code, but I believe the time added by SHP is now only a few seconds to negligible. Test suite portability: - Rewrote test cases whose output depended on platform differences. Specifically, "outside/alarm_error.neg" no longer uses `/bin/cat' as its intentionally erroneous command since its stderr begins with `/bin/cat:' on some platforms but `cat:' on others. - Updated python_server test case output to match python 2.4 so the python_server test cases won't continue to fail on up-to-date systems. Reference manual fixes: - `SHP from a Shell' and `SHP on the Web' now mention that read permission must also be set on a script. The latter also mentions that permissions must instead be set for the web user -- usually with `chmod o+rx'. - As of shp.0.11, there were 6 processing modes. However, the reference manual still said 5 in many places. - Corrected note in `Basic Concepts: SHP Directives' that stated that SHP directives with parameter lists contain only alphabetic characters in their directive symbols. They also contain underscores sometimes. - iterate-namespace documentation: - Improved the explanation of how to use it for embedding SHP variables and values into an embedded language. - Got rid of the example of using it for generating a table. Added note that such uses of it should generally be avoided. Instead, it should be done in the embedded language. - Changed the documentation to explain that order of iteration is now undefined and that any observed order is not guaranteed. - Note that this eliminates todo item 2004/04/12-new-1 from "doc/todo.txt", which was to fully sort by variable name. - Added warning not to use the show-output-if directive since I may remove it in a future release of SHP. - Minor improvements to documentation for variable directives and for processes. Fixed bug in assign directive for namespace copy: - Previously, if the source and destination namespaces overlapped (one contained the other), some destination variables might get the wrong values. - The procedure now, which fixes the problem, is that all source variables are copied before any destination variables are written. Improved efficiency of var and varx directives for large symbol tables: - Previously, they would iterate the symbol table twice: - to check if the specified variable name was defined. - to dereference it. - Now, they do both in one iteration. Error messages for alarm executables and --pipe_output_through flag executables: - I never observed this in practice but did see it as a possibilty in the code. - Previously, they did not always contain consistent error locations. - Specifically, under certain circumstances, the reported error locations described a string of tokenized text rather than the single point at which the parser stopped. They should always describe the single point now. Fixed possible failure during SHP startup: - I never observed this in practice but did see it as a possibilty in the code. - Reorganized main() not to perform any String manipulation before parser__construct(). - Otherwise, any resulting call on yyerror() from String might have failed. ################################################################################ # shp.0.11 ################################################################################ Release Date: 2004/10/25 Internal Tag: shp-0-16 Posted on joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- Added 'use warnings' to perl server. Variable directives: - WARNING: this change is not backward compatible. The variable dereference directive now generates an error page for any variable name that is in the unassigned state. This feature seems to promote cleaner SHP, and it often reveals the kind of bugs that otherwise go unnoticed for a long time. It seems that most modern programming languages exhibit this treatment of undefined variables. - Added variable extract, <$varx>, directive. It's the same as the variable dereference directive, <$var>, except that it never deep copies the value and it leaves the variable undefined. Thus, it's an efficient way to use a potentially large variable value for the last time. - Added default directive, <$default>, which specifies a default value for a variable name: - It takes one parameter, which specifies the variable name, and an optional end parameter, which specifies the default value. Thus, it's similar in format to the assign directive. - Not specifying the end parameter indicates that there is no default value for the variable name. - Regardless of whether the default value is specified, SHP performs no action if the variable name is already assigned. - If the variable name is not already assigned: - If the default value is specified, SHP assigns the default value to the variable name. - If the default value is not specified, SHP generates an error page. Whitespace-collapse directive additions: - The previous forms of the whitespace-collapse directive are now known as the forward-input-whitespace-collapse directive, <$"->, and the reverse-input-whitespace-collapse directive, <$-">. - There are two new forms: the forward-buffer-whitespace-collapse directive, <$'->, and reverse-buffer-whitespace-collapse directive, <$-'>. They are different than the input forms in that, instead of collapsing whitespace in the input stream, they collapse whitespace in the current buffer (which may be the SHP output buffer, a parameter buffer, or the buffer for the replacement text of a reprocess directive). See the reference manual for details. Error messages for malformed pipe names are now more specific. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- User's guide: - I now use the name 'idiom directive' rather than 'shortcut directive' because I believe it better describes the purpose of that set of directives. - Reworded abstract and introduction to better advertise SHP: - Better emphasized idea of mixing scripting languages. - Added discussion of XSLT's disadvantages for templating web pages. - Removed mention of the number of SHP directives. The number no longer makes SHP sound so simple. - A few other rewordings for clarity and correctness. - Fixed a CSS typo that prevented some text from turning blue as it should have. Cygwin installation fixes: - "src/Makefile": - Added $(EXEC_SUFFIX) to 'shp' in 'cp' command in 'install' target. Otherwise, under Cygwin, the 'cp' command fails. - "README.txt": - Added note that 'EXEC_SUFFIX = .exe' in "src/Makefile" should be uncommented for Cygwin. WARNING: these changes are not backward compatible. Changes to stdin: - Previous problems: - Rewinding stdin does not behave as I expected, so the behavior of the stdin or write directive when used multiple times without the end parameter was undesirable: - Stdin cannot be rewound if it is provided by another process. Thus, stdin cannot be rewound for SHP CGI scripts when stdin contains POST data. - Rewinding stdin connected to a terminal causes the user to be prompted for more input. - Rewinding stdin only works as I expected when stdin is a file. - If an input pipe to a process contains too much, it can get full and deadlock SHP. For instance, a file upload may make POST data unpredictably huge. The stdin or write directive that copies the data in SHP's stdin pipe to the child process's input pipe might occur before the talk directive that contains the code that instructs the child process to read its input pipe. In that case, SHP would block in the middle of writing the data to the child process's input pipe once the pipe gets full. Thus, SHP would never get to send the code to the child process to tell it to read its input pipe. Deadlock. - The fix for both of these problems is that SHP is no longer able to copy its stdin pipe data multiple times to child processes' input pipes. Instead, a single process can be designated to inherit SHP's stdin pipe: - The end parameter is now mandatory for the write and stdin directives. - The end parameter is now mandatory for the alarm directive when an executable is specified. - The open directive takes an end parameter that specifies what input pipe should claim SHP's stdin pipe. It may be the process's stdin pipe and must not already be specified in the parameter list. If the input pipe is not named stdin, SHP creates a new file descriptor for it and stores it in SHP_PIPES. This input pipe is immediately considered closed, so it is never writeable with the write or stdin directive. Only one process can be opened this way per invocation of SHP. If it's done again, SHP generates an error page. Now, if a process is connected to SHP's stdin pipe, it can begin reading it immediately rather than waiting for stdin data to be duplicated. - I've created a new open_server_with_stdin directive that's the same as the open_server directive except that it specifies stdin as the end parameter of the open directive. WARNING: these changes are not backward compatible. Fixed problem with pipes becoming full and deadlocking SHP until alarm: - Pipe delimiter change: - The open directive now accepts a pipe delimiter code after the pipe name. Thus, the std pipes can now be specified in the pipe list. It is no longer possible to specify the pipe delimiter code to the read, write, and related idiom directives. See the reference manual for details. - As a result, I modified the open_server, open_server_with_stdin, talk, and close_server directives to accumulate stderr in the variable server_name.stderr with each talk rather than leaving it in the pipe until close_server. - Reader/write threads: - Fixed pipe full problem using a multi-threaded approach. Each specified pipe for a process now has a reader or writer thread to keep the pipe from getting full. The pipe delimiter change mentioned above forces the delimiter to be fixed so that these threads can perform their jobs much more efficiently. - POSIX threads misbehave badly in Cygwin 1.5.11. However, SHP works great with the latest snapshot (2004-Oct-20) of Cygwin... except for thread cancellation, which is necessary for test case "outside/pipe_full_input.pos", which fails only in Cygwin. The problem is that, at the time of this writing, the pthread_cancel() function is not yet fully implemented by Cygwin. The result for SHP users is that, if you fill up an input pipe to a process and the process doesn't unclog the pipe by the time SHP tries to close the process, SHP may deadlock until an alarm activates. This is most likely to occur when the process fails to start or dies unexpectedly. Previously, an unfortunately timed alarm could interrupt code that was modifying some non-local data structure. If that data structure were then accessed by the alarm handler, the results would be undefined. This should no longer be possible except that, on a rare occassion, the error location reported after an SHP alarm may be wrong. See "doc/notes.txt" for details on my implementation of this fix. I found a race condition when opening a process that could have resulted in deadlock immune to alarms. I've never observed this actually occur for certain, but I have observed SHP lock up on very rare occasions. This should no longer be a problem. ################################################################################ # shp.0.10 ################################################################################ Release Date: 2004/07/23 Internal Tag: shp-0-14 Added to joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- The perl server now specifies the 'use strict' pragma. WARNING: This can cause backward compatibility problems for variables whose scopes are not explicitly specified. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- In reference manual: - Formerly, it listed the current SHP release as the release in which <$-> was originally deprecated. It was deprecated as of 0.7. - Formerly, the directive quick reference listed the setenv directive's parameter list with an optional end parameter. It's not optional. - I forgot to update error page output examples according to SHP 0.9 error template changes and removal of hardcoded XHTML. Improvements to distribution procedure in "README.txt". ################################################################################ # shp.0.9 ################################################################################ Release Date: 2004/07/07 Internal Tag: shp-0-13 Added to joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- Added setenv directive. This should make it easier to set environment variables when required by some executables. For instance, perl needs LD_LIBRARY_PATH set when calling libxml from the web. In test suite: - By default, regression test harness only outputs passed: test_case_path.polarity.script for each test case without failures. The old detailed summary is output for a test case only if it has a failure or if the new verbose command-line flag is specified. - Command-line flags are now parsed with perl's Getopt::Long for POSIX conformance plus option bundling. I am now testing SHP primarily with gcc 3.3.2. #------------------------------------------------------------------------------- # 2003/10/06-fix-1 #------------------------------------------------------------------------------- Each process started by SHP is now a process group leader and is killed as such. #------------------------------------------------------------------------------- # 2003/10/11-fix-1 #------------------------------------------------------------------------------- XHTML elements (br, ol, li, pre, and p) and entities ( ) are no longer used for formatting since the error page template now encloses all text in a pre element. Thus, SHP is no longer be hardcoded for XHTML except in the templates, which can be rewritten by the user. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Fixed two pipe problems that resulted in deadlock: - Formerly, in order to send EOF across a child process's input pipes, all child processes that were opened after it had to be closed first. The problem was that all those later child processes inherited from the parent all open pipe file descriptors for the first child process. Thus, its input pipes couldn't be closed from the parent. Now, fcntl() with FD_CLOEXEC sets the file descriptors to close automatically on exec in all the later child processes. - Formerly, a child process could not send EOF across its std output pipes before terminating. The problem was that the original file descriptors associated with the pipes were not closed after dup2(). Formerly, scanner's line number was not incremented for a newline appearing as the whitespace character in an opening processing-instruction directive. Error messages: - Added check that putenv() returns without error when called from the open directive for SHP_PIPES. Added the same check when putenv() is called from the new setenv directive. - Updated some very out of date waitpid()-related error message that referred to a process as a command spawn. - Unclosed process lists and param group lists in error pages are now bulleted with 'o', which is often used as a bullet in UNIX command-line usage messages. These lists used to be just indented instead. (Param scenario lists are still numbered.) This is easier to read. - When SHP reports a list of unclosed processes in the current scope, the list is now alphabetized. - In error templates: - XHTML error page template now places message in pre element rather than p element - UTF-8 -> iso-8859-1 since this seems to be the standard now. In user guide: - In tutorial, reworded explanation of POST data. Formerly, it said that "pipes are not copied to spawned processes". I added "under SHP" since this is not true in general. - Corrected exit status explanation in reference manual. Exit status is always non-zero when SHP generates an error page but not necessarily when it generates an error message (for instance SIGKILL errors on stderr). - Previously, SHP_PIPES explanation did not mention that pipe file descriptors are listed in the same order as their respective names in the open directive. Also, it didn't specify that SHP_PIPES is defined in the child process but not in the SHP parent process. - Various other more minor corrections and rewordings for clarity. - Moved xml declarations to the top of all XHTML files as required by the W3C validators and the XML standard. - UTF-8 -> iso-8859-1 since this seems to be the standard now. - IE doesn't like XML comments above opening html tag when there's an XML processing instruction. Mozilla doesn't care, as per the XML standard, but I fixed it anyway. In test suite: - Set up of SHP-install-directory directive test case output should be less confusing now since I've removed the trailing '/asdf'. Improvements to distribution procedure in "README.txt". ################################################################################ # shp.0.8 ################################################################################ Release Date: 2004/05/10 Internal Tag: shp-0-12 Added to joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- The old 'Reference Manual', which used to be stored in "doc/manual/", is now a subpage of the new 'User Guide', which is stored in "doc/user_guide/". The user guide includes a very brief tutorial that should get you up and running using perl or python. "bin/" now contains a python server script. Both the tutorial and the reference manual explain its usage as well as the usage of the perl server script. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Previously, the write directive (and all shortcut directives that use it) didn't check that the specified pipe was open. ################################################################################ # shp.0.7 ################################################################################ Release Date: 2004/04/26 Internal Tag: shp-0-11 Added to joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- SHP can now compile under GNU/Linux or Cygwin. The whitespace-collapse directive now takes two forms. The forward form, '<$"->', collapses the following whitespace. It is equivalent to the now deprecated form '<$->'. The reverse form, '<$-">', collapses preceding whitespace. The addition of the second form makes pretty-formatting parameter lists much easier. The directive-escape directive now takes two forms: '<$>' and '<$$'. Both have the same replacement text. The first looks best for generating '<$' in most contexts. However, the second is much more readable when escaping actual SHP directives. The close directive now takes an optional second parameter. When specified, it names a single input pipe to close for the process specified by the first parameter. Thus, it's now possible to send EOF on a pipe without closing the process. When SHP can't kill an unclosed process at termination, it now writes a message to stderr naming the process and giving its pid. Some improvements to the installation procedure notes in "README.txt". #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Fixed compilation errors for some platforms: - Converted all C++ style comments to C style for compatibility with ISO C90. ISO C99 support isn't widespread enough. This change makes compilation messages much easier to read for recent versions of gcc. - A C-string literal in "shp.c" was too long for ISO C90, so I broke it apart. - Added missing #include to "Pipe.c" and "scanner.flex". When closing a process, SHP previously reported an error closing output pipes as an error closing input pipes. Stopped processes were previously not detected correctly by the close directive. In a few rare cases (involving nested processing-instruction directives or an unclosed multiline-comment directive), location information was not reported correctly in error messages. ################################################################################ # shp.0.6 ################################################################################ Release Date: 2004/04/12 Internal Tag: shp-0-10 Added to joeldenny.com. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- The iterate-namespace directive may now take a second parameter group to restrict the iteration to a specific range of sub-namespace levels. The iterate-namespace directive is now guaranteed to iterate ascendingly through all namespace variables that are scoped to the same input stream as the zeroth sub-namespace level. (The position and order of all other variables in the iteration is undefined. In a future version of SHP, they may be sorted in with the others.) #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- Fixed namespace operations (assign and iterate-namespace). Formerly, they only acted on variables that belonged in the same input stream as the 0th sub-namespace level. That is, they didn't act on any (deeper) sub-namespace level that belonged in an input stream enclosing that input stream. Removed the statement from the manual that claimed that only the open directive takes a parameter group. There were others as of 0.5. ################################################################################ # shp.0.5 ################################################################################ Release Date: 2004/03/27 Internal Tag: shp-0-8 Installed for Dr. Brad Meyer's group in Clemson University's physics dept. #------------------------------------------------------------------------------- # New. #------------------------------------------------------------------------------- Added the show-output-if directive. See the manual. Added '--version' and '--help' flags to SHP executable. Added '--' flag to indicate that the next argument is the filename for those cases where the filename begins with '-'. Added command-line flag '--pipe_output_through=cmd', which, after SHP successfully processes the entire document, pipes SHP's output buffer through 'cmd'. Added an example to the manual for the replace directive, partially completing todo item 2003/11/11-new-1. #------------------------------------------------------------------------------- # Fix. #------------------------------------------------------------------------------- NOT BACKWARDS COMPATIBLE: The replace directive now accepts multiple param groups with 2 params each rather than just an even number of params in only one param group. This seems cleaner conceptually, and it makes the directive slightly easier to read. NOT BACKWARDS COMPATIBLE: The SHP executable exit status in the case of all alarm activations is now non-zero to indicate a failure. It used to be zero in certain alarm activation cases. Added section to manual about SHP executable exit status. Fixed command-line argument trouble with SHP executable. When the query string contains no '&' or '=', the web server passes the entire query string to a CGI program as an additional command-line argument. Previously, SHP reported a usage message in that case. Now, it ignores command-line arguments after the filename. ################################################################################ # shp.0.4 ################################################################################ Release Date: 2004/03/25 Internal Tag: shp-0-7 Initial public release. Installed for Dr. Brad Meyer's group in Clemson University's physics dept.