Simplified lookup of property values

[mnieper]

The property-value procedure, which can be used within the dynamic extent of the expander's call to a macro transformer, looks up the value of a property of an identifier in the macro's use environment. This obviates the need to follow a specific transformer protocol to obtain a lookup procedure that returns property values and lets macro helper functions look up property values independently.

[mflatt]

Did you consider using with-continuation-mark instead of parameterize?

In an attempt to measure the expansion overhead of parameterize, I tried

(define-syntax (m stx)
  (syntax-case stx ()
    [(_ n)
     (if (eqv? (datum n) 0)
         #'"ok"
         #`(m #,(- (datum n) 1)))]))

(time
 (eval '(m 1000000)))

The implementation here was more than 10% slower to evaluate (m 1000000). When I tried using with-continuation-mark in place of the parameterize, it was about 1% slower. The lookup side will be significantly slower via a continuation mark than a thread-context lookup, but it won't be all that slow, and property-lookup calls seem likely to be rare. Also, no changes would be needed to the thread context.

[mnieper]

Thank you very much for your prompt and thorough review! There was no particular reason for why I used parameterize only that the existing code base of Chez doesn't make use of continuation marks yet.

I pushed a version that uses continuation marks; it remains to make bootstrapping with an old version of Chez that does not know about continuation marks work.

PS The commits below enable bootstrapping again.

[jltaylor-us]

A procedure that is only allowed to be called during expansion does not seem very Chez-like. Is there anything else in Chez that has any analogous restriction?

[mnieper]

Well, the lookup procedure of the old protocol implicitly had the same restriction.

I changed the code so that property-value can also be called outside the dynamic extent of a macro transformer. A programmer that makes use of this feature has to be aware about phasing issues. The new tests added at the end of the property-value mat in 8.ms show how this acts. Thanks to Chez Scheme's implicit phasing, it simply works when importing from a library or when working in the top-level environment.

[dpk]

In the R7RS large draft this procedure is called identifier-property but its signature and behaviour are not yet finalized.

[mnieper]

In the R7RS large draft this procedure is called identifier-property but its signature and behaviour are not yet finalized.

I know, but property-value is semantically more correct and also in line with the naming of Chez's existing compile-time-value-value.

[burgerrg]

I'm hesitant to do something that causes macro expansion to slow down, even by 1%. It would help me to see an example where this procedure makes the macro much simpler and would justify the cost.

[mnieper]

I'm hesitant to do something that causes macro expansion to slow down, even by 1%. It would help me to see an example where this procedure makes the macro much simpler and would justify the cost.

For example, with the latest version, you can query identifier properties on the right-hand side of define-property or meta define, for example, which makes perfect sense. You cannot do so directly with the currently existing model in Chez Scheme.

In any case, in one sense, the slowdown is negligible. Chez Scheme doesn't have a continuation barrier (à la Racket) in the expander, so reinstatiation of continuations captured in a macro transformer call long after that call has ended is possible. Through this, it becomes observable that Chez's expander uses mutable state. Moreover, one can cause the expander to produce nonsensical results. So, at one point in the future, Chez should get a continuation barrier (which would have to be placed more or less where my patch currently adds a continuation mark), and compared with the barrier, the continuation mark would be truly negligible.

On the other hand, if we just declare that trying to jump back into transformer calls through call/cc is unsafe (but this may possibly violate the R6RS safety guarantee), I could operate just with set!, which would be the fastest. But I would really not want to do this because that is not future-safe, should a limited way of non-local control for macro transformers ever be implemented or guaranteed.

[mnieper]

I forgot to add that the slowdown of the macro expansion will be far less than 1% in practice. Matthew's example macro basically did nothing, so it was able to mostly measure the speed of the expander itself. Most macros encountered in practice do something non-trivial. A good measure in practice would be measuring how long it takes to expand some file with non-trivial macro use, e.g., s/cpnanopass.ss.

[burgerrg]

The manual entry for define-property describes the macro get-property that seems to be very similar to what you're proposing. Would it work for you?

[mnieper]

The manual entry for define-property describes the macro get-property that seems to be very similar to what you're proposing. Would it work for you?

No, get-property is not the same, and not similar. get-property can inject a property value (suitably wrapped in a syntax object) during expansion of a piece of code. property-value, on the other hand, delivers the property value while code is executing.

[burgerrg]

A code example would help me understand what you're trying to accomplish.

[mnieper]

A code example would help me understand what you're trying to accomplish.

For example,

(define-property id1 key1 (property-value #'id2 #'key2))

defines a property that copies the property value from another property. Replacing property-value with get-property would not work.

A different use case is that of a macro transformer helper procedure that needs to access property values. With the existing model implemented in Chez Scheme, it needs a collaborating macro transformer because that has to acquire the lookup procedure. This can lead to a program style that couples components in a stronger way than it should and where lookup arguments have to be passed down.

For example, a library API implementing some abstraction over the property facility should work seamlessly without having to ask the user code to acquire the lookup procedure and make it available.

[burgerrg]

Here's your example in code:

(define-syntax get-property
  (lambda (x)
    (lambda (r)
      (syntax-case x ()
        [(_ id key)
         #`'#,(datum->syntax #'* (r #'id #'key))]))))
(let ()
  (define id2 'id2)
  (define key2 'key2)
  (define id1 'id1)
  (define key1 'key1)
  (define-property id2 key2 'hello)
  (define-property id1 key1 (get-property id2 key2))

  (printf "(get-property id2 key2) = ~s\n" (get-property id2 key2))
  (printf "(get-property id1 key1) = ~s\n" (get-property id1 key1)))

This prints:

(get-property id2 key2) = hello
(get-property id1 key1) = hello

[soegaard]

Is local-expand from Racket a relevant real-world example in this context?

The ultra-short explanation:

Expands stx in the lexical context of the expression currently being expanded.

The documentation notes that:

This procedure must be called during the dynamic extent of a syntax transformer application by the expander or while a module is visited (see syntax-transforming?), otherwise the exn:fail:contract exception is raised.

Searching for extent on https://docs.racket-lang.org/reference/stxtrans.html reveals a few other procedures that can only be called during expansion.

[mnieper]

Here's your example in code:

(define-syntax get-property
  (lambda (x)
    (lambda (r)
      (syntax-case x ()
        [(_ id key)
         #`'#,(datum->syntax #'* (r #'id #'key))]))))
(let ()
  (define id2 'id2)
  (define key2 'key2)
  (define id1 'id1)
  (define key1 'key1)
  (define-property id2 key2 'hello)
  (define-property id1 key1 (get-property id2 key2))

  (printf "(get-property id2 key2) = ~s\n" (get-property id2 key2))
  (printf "(get-property id1 key1) = ~s\n" (get-property id1 key1)))

This prints:

(get-property id2 key2) = hello
(get-property id1 key1) = hello

Thank you for the example. If the code is part of a library, the library will at some point be expanded. At this point, the macro use get-property will be replaced by the quotation of the value of the property at that time. This is a constant, so when the library is visited later, the property of id1 will be associated with that constant value. On the other hand, when property-value is used, the value will be retrieved at visit-time of the library.

This makes no difference for types like symbols, but it does make a difference when a procedure, hashtable, or some other compound type is the value of the original property. For example, imagine 'hello is replaced by (current-output-port). If get-property is used, at visit-time, id2 would hold the then current output port as a property value, but id1 would receive the stale output port that was current at expand-time.

And this only works as long as Chez Scheme allows arbitrary objects in quoted contexts, which is not part of R6RS (there, syntax objects are only (partially) wrapped Scheme datums). Is the latter anywhere documented in CSUG, or is it just undefined behaviour?

[mnieper]

Is local-expand from Racket a relevant real-world example in this context?

The ultra-short explanation:

Expands stx in the lexical context of the expression currently being expanded.

The documentation notes that:

This procedure must be called during the dynamic extent of a syntax transformer application by the expander or while a module is visited (see syntax-transforming?), otherwise the exn:fail:contract exception is raised.

Searching for extent on https://docs.racket-lang.org/reference/stxtrans.html reveals a few other procedures that can only be called during expansion.

The Racket version of property-value seems to be syntax-local-value. If you look at the Nanopass source code for Racket, you can see this used while in the Chez Scheme version, the lookup procedure, there called rho, has to passed down as an argument to the final consumers.

Racket may have to be stricter about the time procedures are allowed to be called because of the strictly separated phases there. Chez Scheme, with its implicit phasing model, is different, so not every Racket restriction should apply to Chez Scheme.

[owaddell]

Like Bob, I'm having some trouble understanding the problem this PR aims to address. It appears the aim is to avoid having to request a lookup procedure and pass it in to other help procedures used by the transformer.

If so, perhaps interested parties can do this without modifying Chez Scheme by using something like the following:

(library (scheme-alt)
  (export property-value syntax-case)
  (import (rename (scheme) (syntax-case real-syntax-case)))
  (export (import (except (scheme) syntax-case)))

  (define *do-lookup* (make-parameter (lambda (id key) #f)))

  (define (property-value id key) ((*do-lookup*) id key))

  (define-syntax syntax-case
    (syntax-rules ()
      [(_ expr aux clause ...)
       (lambda (lookup)
         (parameterize ([*do-lookup* lookup])
           (real-syntax-case expr aux clause ...)))]))
  )

I imagine this could use Matthew's continuation mark alternative instead. Here is a pared-down example where a transformer calls a help procedure that uses the exported syntax-case and property-value.

(let ()
  (import-only (scheme-alt))
  (define joy)
  (define-property = joy "happy")
  (meta define (three x) (property-value x #'joy))
  (define-syntax (bar x)
    (syntax-case x ()
      [(_ y) (three #'y)]
      [(_ x y) (property-value #'x #'y) "hit"]
      [_ "miss"]))
  (pretty-print (list (bar =) (bar +) (bar = joy) (bar and joy))))

[mnieper]

Thank you for your code, Oscar. Of course, by importing customised versions of R6RS procedures or syntax, one can get rid of having to handle lookup directly - although the correct place would be, I think, to customise define-syntax, let-syntax and letrec-syntax because syntax-case has other uses than as an outer form of a macro transformer and because not every macro transformer has a syntax-case as an outer form. Customising define-syntax, etc., however, has the problem that it would break variable transformers.

In any case, the solution you propose has (at least) two shortcomings compared to the solution this PR offers:

1. As a library writer of syntax and procedures that internally make use of lookup, I would have to ask the users of the library to use special protocols at unrelated places (e.g. by using my syntax-case or my version of define-syntax or my version of make-foo-transformer) to make the API usable. While this is unpleasant at best, it becomes a real problem if the consumer of my library doesn't have direct access to the transformer procedure because that comes from another abstraction coming from another library. It also becomes a real problem if another library that has to cope with the same problem wants the user to use make-bar-transformer. In other words, we have a composition problem.
2. The second shortcoming is that it doesn't make property values available to meta definitions and the right-hand sides of define-property (or define-syntax) during visit-time of a library. In fact, in your second code block, there would be a problem if the code is part of the top-level of a library when that library is visited, and the first use of property-value would not be in a lambda.

Let me add that I appreciate this discussion. A significant part of Chez Scheme's appeal comes from its relative conservatism, so the addition of features needs to be well-considered. In the case of property-value, however, I believe that the benefits outweigh possible drawbacks. The only drawback seems to be a minor slowdown of macro expansion, which is very likely only measurable in non-realistic benchmarks like Matthew's example. Moreover, as I have sketched above, making Chez's macros robust against unlimited use of call/cc by the user will need some kind of mark in the call stack anyway.