cisco
Unhelpful error message
I am currently stymied on the project I am working on by this:
> compiling newcash.scm with output to newcash.so
compiling libraries/gtk.scm with output to libraries/gtk.so
compiling account.scm with output to account.so
compiling definitions.scm with output to definitions.so
compiling utilities.scm with output to utilities.so
compiling book.scm with output to book.so
>
scheme < build-whole-program.scm
Chez Scheme Version 9.5.1
Copyright 1984-2017 Cisco Systems, Inc.
>
Exception in process-ir!: encountered bare lsrc item while processing wpo file (begin (($primitive 3 $import-library) (quote (...)) (quote ()) (quote scheme)) (($primitive 3 $sc-put-cte) (quote #[...]) (quote (...)) (quote *top*)))
>
make[1]: Leaving directory '/home/dca/Software/newcash_scheme/newcash'
dca@franz:~/Software/newcash_scheme$
I am generating .wpo files in the compilation of individual files and build-whole-program.scm looks like this:
(begin
(compile-whole-program "newcash.wpo" "newcash.so")
(make-boot-file "newcash.boot" '("petite") "newcash.so")
)
The error gives me nothing useful to use to try to fix this, so I can't proceed with this work. This feels like a bug in Chez to me, so if there's anything I can do to help debug this, please let me know. If I'm missing something here, please let me know that too!
It looks like you've compiled the individual source files with compile-file when you want to use compile-program.
Also, it seems weird to turn a rnrs program into a boot file. You can run the program with the --program switch on the command line.
On Wed, 6 Mar 2019 at 12:03, Graham Watt [email protected] wrote:
It looks like you've compiled the individual source files with compile-file when you want to use compile-program.
Yes, that works, but I note that until I made some recent changes, compile-file worked without issue. Same setup, no problem. Even in the build I reported in my original message, prior to the part I included in the message, three other applications got compiled and boot-files built exactly the same way, using compile-file without a problem.
Also I don't get the sense from the documentation that compile-program is necessary when compiling top-level programs, only that things will run a bit faster than if compile-file is used. So I would like to understand whether what I did was an error and, if so, would request that the documentation be improved to say so. If using compile-program is just a performance optimization, then the documentation is fine the way it is.
Lastly, I think some attention needs to be given to not producing the error I encountered. If what I did is considered an error by the system, then it should detect it and provide an informative error message. If using compile-file is allowable in this case, but perhaps not optimal, then the appearance of this error still seems to me to be a bug.
Also, it seems weird to turn a rnrs program into a boot file. You can run the program with the --program switch on the command line.
See the Building and Running the Applications subsection of Section 2.8 of the Chez Manual. There's a whole discussion there of the advantage of using boot files. That's why I'm doing this.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
As I understand it, Chez Scheme operates two with different semantics. I am going to call these semantics R6RS and REPL[1]. R6RS semantics include programs and libraries, while REPL semantics include files being loaded with the load procedure, scripts, and the REPL[2]. In your specific case, compile(-whole)-program compiles the source with R6RS semantics and compile-file compiles the source with REPL semantics. The result of compile-file unsuitable for use by compile(-whole)-program, even if the source being compiled is syntactically a valid R6RS program. This may be confusing because any and all scheme sources are valid for consumption by compile-file, including R6RS programs and libraries. You must choose which semantics you want
Let's talk about boot files. When making boot files, any source files specified are compiled with REPL semantics and not R6RS semantics. Any object files keep their semantics, and Chez Scheme appears flexible enough to mix and match the different semantics when it expects REPL mode.
It looks like you've compiled the individual source files with compile-file when you want to use compile-program.
Yes, that works, but I note that until I made some recent changes, compile-file worked without issue. Same setup, no problem. Even in the build I reported in my original message, prior to the part I included in the message, three other applications got compiled and boot-files built exactly the same way, using compile-file without a problem.
Using compile-file to generate intermediate files for make-boot-file is completely acceptable, but using compile-whole-program on the output of compile-file is not.
Also I don't get the sense from the documentation that compile-program is necessary when compiling top-level programs, only that things will run a bit faster than if compile-file is used.
If by "top-level program" you mean R6RS program, then compile-program is absolutely needed. If you instead mean "code that runs upon Chez Scheme starting" then you are correct, and compile-program is not needed. However, compile-program is different from just a faster version of compile-file.
Also, it seems weird to turn a rnrs program into a boot file. You can run the program with the --program switch on the command line.
See the Building and Running the Applications subsection of Section 2.8 of the Chez Manual. There's a whole discussion there of the advantage of using boot files. That's why I'm doing this.
I was unclear. I was not questioning making and distributing boot files, but rather questioning creating boot files from R6RS programs. Both boot files and R6RS programs are perfectly valid choices for distribution, but probably not intermingled. If you want to continue in making boot files, I think there's an easier way than what you've been doing:
(make-boot-file "my-boot-file" '("petite") "list" "of" "source" "files")
make-boot-file will handle compilation of source files as needed. You don't need to compile them beforehand.
If you decide to use R6RS program instead, there's also an easier way.
(compile-imported-libraries #t)
(generate-wpo-files #t)
(compile-program "program-source")
(compile-whole-program "program-source.wpo" "whole-program-output")
Combining compile-imported-libraries and generate-wpo-files generates wpo files for all libraries imported by your program so that you don't have to compile each library file individually.
After this, the file "whole-program-output" is now eligible for running with the --program command line flag.
[1] REPL mode operates by running each top-level expression through eval, whether it's executing source code or object code. Additionally, unknown identifiers are looked up in the environment via top-level-value which allows identifiers to be created at runtime via set-top-level-value!. R6RS mode does not run each top-level expression through eval. Every top-level expression is run as a single expression as though nested inside a begin, with the added restriction that all identifiers must be lexically visible; no runtime environment lookups. The different semantics between REPL and R6RS modes allow the compiler to generate more optimal code for R6RS programs and libraries than can be achieved in REPL mode
[2] R6RS mode:
- compile-program / compile-whole-program
- compile-library / compile-whole-library
REPL mode:
- compile-file
- compile-script
- load (You can use
loadon compiled R6RS programs or libraries at which point they maintain R6RS semantics)
On Fri, 15 Mar 2019 at 09:25, Graham Watt [email protected] wrote:
As I understand it, Chez Scheme operates two with different semantics. I am going to call these semantics R6RS and REPL[1]. R6RS semantics include programs and libraries, while REPL semantics include files being loaded with the load procedure, scripts, and the REPL[2]. In your specific case, compile(-whole)-program compiles the source with R6RS semantics and compile-file compiles the source with REPL semantics. The result of compile-file unsuitable for use by compile(-whole)-program, even if the source being compiled is syntactically a valid R6RS program. This may be confusing because any and all scheme sources are valid for consumption by compile-file, including R6RS programs and libraries. You must choose which semantics you want
Let's talk about boot files. When making boot files, any source files specified are compiled with REPL semantics and not R6RS semantics. Any object files keep their semantics, and Chez Scheme appears flexible enough to mix and match the different semantics when it expects REPL mode.
It looks like you've compiled the individual source files with compile-file when you want to use compile-program. Yes, that works, but I note that until I made some recent changes, compile-file worked without issue. Same setup, no problem. Even in the build I reported in my original message, prior to the part I included in the message, three other applications got compiled and boot-files built exactly the same way, using compile-file without a problem.
Using compile-file to generate intermediate files for make-boot-file is completely acceptable, but using compile-whole-program on the output of compile-file is not.
But in my experience, it works 99% of the time!
Also I don't get the sense from the documentation that compile-program is necessary when compiling top-level programs, only that things will run a bit faster than if compile-file is used.
If by "top-level program" you mean R6RS program, then compile-program is absolutely needed. If you instead mean "code that runs upon Chez Scheme starting" then you are correct, and compile-program is not needed. However, compile-program is different from just a faster version of compile-file.
Yes, I am talking about r6rs top-level programs, with an 'import' declaration and the only top-level definition being (define main). And I repeat -- this worked, in the sense that the programs both compiled and ran, compiling them with 'compile-file'. It was only after making some changes that I encountered the failure that I reported in my original post. And prior to that failure, several other applications got built using compile-file and the boot files got generated normally. And I repeat again, the documentation does not say, at least to me, that "compile-program is absolutely needed". Perhaps I have mis-read it, and I haven't looked at it since the day I reported this error, because I have moved on to working on other things. But my point is that I don't understand your statement that "compile-program is absolute needed". I have many examples of compile-file working just fine with R6RS top-level programs. For me, the essential issue is that this needs to be clarified in the documentation, whatever the truth is, and the system should never produce an error such as the one I encountered.
Also, it seems weird to turn a rnrs program into a boot file. You can run the program with the --program switch on the command line. See the Building and Running the Applications subsection of Section 2.8 of the Chez Manual. There's a whole discussion there of the advantage of using boot files. That's why I'm doing this.
I was unclear. I was not questioning making and distributing boot files, but rather questioning creating boot files from R6RS programs. Both boot files and R6RS programs are perfectly valid choices for distribution, but probably not intermingled. If you want to continue in making boot files, I think there's an easier way than what you've been doing:
(make-boot-file "my-boot-file" '("petite") "list" "of" "source" "files")
make-boot-file will handle compilation of source files as needed. You don't need to compile them beforehand. If you decide to use R6RS program instead, there's also an easier way.
(compile-imported-libraries #t) (generate-wpo-files #t) (compile-program "program-source") (compile-whole-program "program-source.wpo" "whole-program-output")
Combining compile-imported-libraries and generate-wpo-files generates wpo files for all libraries imported by your program so that you don't have to compile each library file individually.
Yes, I've been doing the latter -- using compile-imported-libraries and generate-wpo-files both set true.
After this, the file "whole-program-output" is now eligible for running with the --program command line flag.
This discussion, since the paragraph beginning "I was unclear" is very useful because, again, I got none of this from the documentation, at least as much of it as I have read. Perhaps it is there and I just haven't read enough of it.
Thanks very much for the clarifications.
/Don
[1] REPL mode operates by running each top-level expression through eval, whether it's executing source code or object code. Additionally, unknown identifiers are looked up in the environment via top-level-value which allows identifiers to be created at runtime via set-top-level-value!. R6RS mode does not run each top-level expression through eval. Every top-level expression is run as a single expression as though nested inside a begin, with the added restriction that all identifiers must be lexically visible; no runtime environment lookups. The different semantics between REPL and R6RS modes allow the compiler to generate more optimal code for R6RS programs and libraries than can be achieved in REPL mode
[2] R6RS mode:
- compile-program / compile-whole-program
- compile-library / compile-whole-library REPL mode:
- compile-file
- compile-script
- load (You can use load on compiled R6RS programs or libraries at which point they maintain R6RS semantics)
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cisco/ChezScheme/issues/405#issuecomment-473284603, or mute the thread https://github.com/notifications/unsubscribe-auth/AEzjJ5XCBf4V8EPXHxH5d4quJpQrA4bSks5vW59TgaJpZM4bhT9Y .
I agree the error message is a bit unhelpful, and I've tried to improve it a bit in commit 11cee34. It will now report:
expected program or library form, but encountered top-level expression
<top-level expression> processing wpo file <wpo file name>
where the full message is essentially on one line, I've just wrapped it here to avoid needing to scroll it within the text box.
Essentially, compile-whole-library and compile-whole-program work by reading in an intermediate representation from the wpo files looking for library and program information. It is possible for any wpo file built from source that contained top-level expressions. It is worth noting that any source file that contains top-level expressions can produce a wpo file with top-level expressions, unless it is compiled with compile-program.
@gwatt is correct that there are essentially two modes for the compiler, unfortunately, these modes are less related to how code is compiled than it is to which mode the reader is when it starts up, with the exception of compile-program. Both compile-library, compile-program, and their maybe- prefixed equivalents start the reader in r6rs reader mode, while compile-file and its maybe- equivalent, compile-script, compile-port, compile-to-port, and compile-to-file all start in chezscheme reader mode. This mode can be changed using the #r6rs or #chezscheme directive within the file.
With the exception compile-program and maybe-compile-program, all the other versions of compile-* read an expression from the file, compile the expression, and then read and compile the next expression. In general, the expectation is that in compile-library this will contain one or more library forms, but it is not required or checked. compile-file called on a file that contains only a library form will produce the same results (modulo changes caused by the differences in reader mode) as compile-library.
compile-program on the other hand, reads all of the expressions in the file into a single list of expressions which it wraps in a top-level-program form. So compiling with compile-program and compile-file will create different results. compile-file called on a program will produce a series of one or more top-level expressions.
Up until recently, R6RS library and program forms were not compatible with make-boot-file which was a tool that predated R6RS support in Chez Scheme. The make-boot-file support has been improved and now allows creating boot files with libraries and programs. There is some advantage to combining these two things. compile-whole-program and compile-whole-library essentially combine the source code of a set of libraries (with a program in the case of compile-whole-program) and produces a combined internal representation form that the compiler can compile as a single chunk of code, allowing increased opportunities for optimization across library and program boundaries, since these boundaries are effectively removed. make-boot-file on the other hand combines one or more binary files (or compiles each individually if they are listed as source files) with a header that indicates which other boot files are needed to load before this boot file. This allows the result to be handed to the scheme executable with -b and should cause the appropriate boot files to be auto-loaded as part of loading the boot file.
So the two do not really overlap, though they did not always play nice together.
Anyway, hopefully this addresses this issue.
On Tue, 19 Mar 2019 at 22:58, Andy Keep [email protected] wrote:
I agree the error message is a bit unhelpful, and I've tried to improve it a bit in commit 11cee34 https://github.com/cisco/ChezScheme/commit/11cee34502470d720d611ffd0799353e8663a7f1. It will now report:
expected program or library form, but encountered top-level expression
processing wpo file What is a "program" form? Section 10.3 of "The Scheme Programming Language" document: "A top-level program is not a syntactic form per se ..."
If possible, I would suggest that the error be more specific about what the user did wrong and how to correct it, since there's a dizzying array of compile-* choices, and it is still somewhat unclear what to use when, though what you wrote below if definite progress toward clarifying this issue.
where the full message is essentially on one line, I've just wrapped it here to avoid needing to scroll it within the text box.
Essentially, compile-whole-library and compile-whole-program work by reading in an intermediate representation from the wpo files looking for library and program information. It is possible for any wpo file built from source that contained top-level expressions. It is worth noting that any source file that contains top-level expressions can produce a wpo file with top-level expressions, unless it is compiled with compile-program.
@gwatt https://github.com/gwatt is correct that there are essentially two modes for the compiler, unfortunately, these modes are less related to how code is compiled than it is to which mode the reader is when it starts up, with the exception of compile-program. Both compile-library, compile-program, and their maybe- prefixed equivalents start the reader in r6rs reader mode, while compile-file and its maybe- equivalent, compile-script, compile-port, compile-to-port, and compile-to-file all start in chezscheme reader mode. This mode can be changed using the #r6rs or #chezscheme directive within the file.
With the exception compile-program and maybe-compile-program, all the other versions of compile-* read an expression from the file, compile the expression, and then read and compile the next expression. In general, the expectation is that in compile-library this will contain one or more library forms, but it is not required or checked. compile-file called on a file that contains only a library form will produce the same results (modulo changes caused by the differences in reader mode) as compile-library.
compile-program on the other hand, reads all of the expressions in the file into a single list of expressions which it wraps in a top-level-program form. So compiling with compile-program and compile-file will create different results. compile-file called on a program will produce a series of one or more top-level expressions.
Up until recently, R6RS library and program forms were not compatible with make-boot-file which was a tool that predated R6RS support in Chez Scheme. The make-boot-file support has been improved and now allows creating boot files with libraries and programs. There is some advantage to combining these two things. compile-whole-program and compile-whole-library essentially combine the source code of a set of libraries (with a program in the case of compile-whole-program) and produces a combined internal representation form that the compiler can compile as a single chunk of code, allowing increased opportunities for optimization across library and program boundaries, since these boundaries are effectively removed. make-boot-file on the other hand combines one or more binary files (or compiles each individually if they are listed as source files) with a header that indicates which other boot files are needed to load before this boot file. This allows the result to be handed to the scheme executable with -b and should cause the appropriate boot files to be auto-loaded as part of loading the boot file.
This is a good start, in my opinion, toward something that could be included in the Chez manual, to help the user make proper choices re building an executable program given how the program is structured. I think such an explanation is essential, because this is pretty bewildering as it currently stands. And it's in conflict with why many of us use Scheme -- simplicity and clarity.
So the two do not really overlap, though they did not always play nice together.
Anyway, hopefully this addresses this issue.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cisco/ChezScheme/issues/405#issuecomment-474664819, or mute the thread https://github.com/notifications/unsubscribe-auth/AEzjJxFGXSog2sUfI9lZhLW6Q5FSfEWQks5vYaPTgaJpZM4bhT9Y .
If possible, I would suggest that the error be more specific about what the user did wrong and how to correct it, since there's a dizzying array of compile-* choices, and it is still somewhat unclear what to use when, though what you wrote below if definite progress toward clarifying this issue.
That is what this error message attempts to do. It is not possible for compile-whole-program and compile-whole-library to know how you produced the wpo file from your source file, so it isn't really possible to be "more specific" about how the top-level expression got into the wpo file. For instance, it is possible you compiled your source with compile-library but your source file had a top-level expression in it (possibly in addition to the library form), an error message that recommends a user use compile-library is simply going to be frustrating (and erroneous).
I'd be happy to add to the documentation around this---including providing some aid on how this error message should be interpreted.
On Wed, 20 Mar 2019 at 08:59, Andy Keep [email protected] wrote:
If possible, I would suggest that the error be more specific about what the user did wrong and how to correct it, since there's a dizzying array of compile-* choices, and it is still somewhat unclear what to use when, though what you wrote below if definite progress toward clarifying this issue.
That is what this error message attempts to do. It is not possible for compile-whole-program and compile-whole-library to know how you produced the wpo file from your source file, so it isn't really possible to be "more specific" about how the top-level expression got into the wpo file. For instance, it is possible you compiled your source with compile-library but your source file had a top-level expression in it (possibly in addition to the library form), an error message that recommends a user use compile-library is simply going to be frustrating (and erroneous).
I suspected that might be the case, which is why I said "If possible ...". I understand the problem.
I'd be happy to add to the documentation around this---including providing some aid on how this error message should be interpreted.
I think it's pretty important to clear this up, because it's in series with producing a usable program.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cisco/ChezScheme/issues/405#issuecomment-474817883, or mute the thread https://github.com/notifications/unsubscribe-auth/AEzjJwsNYgTu9sqZCUqI8y6cp9GHqxgpks5vYjC0gaJpZM4bhT9Y .