CFFI Support

[Shinmera]

CFFI is one of the most used and most central libraries of the CL ecosystem. Adding support for it in Clasp is therefore vital for Clasp's ability to demonstrate itself as a useful, new implementation.

[drmeister]

frgo has expressed interest in adding CFFI to Clasp. He's investigating it.

[Shinmera]

I don't want to release until @dg1sbg has at least basic CFFI support in. Please let me know when that is ready for testing.

[drmeister]

frgo has done an absolutely fantastic job incorporating CFFI.
cl-opengl works as of a couple of days ago including callbacks. frgo is doing some more testing and there is some cleanup that we need to do to remove translation functions that turned out not to be needed.

[drmeister]

There are still some CFFI tests that are failing. I can tackle them now that I've got the calling convention sorted out again.

[kpoeck]

(load "~/quicklisp/setup.lisp")
 (time (ql:quickload "cffi" :verbose t))
 (time (ql:quickload "cffi-tests" :verbose t))
(cffi-tests::run-all-cffi-tests)
->
....
29 out of 323 total tests failed: FUNCALL.CHAR, DEFCFUN.CHAR, DEFCFUN.NOOP, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.CHAR, CALLBACKS.LONG, CALLBACKS.BFF.1, CALLBACKS.BFF.2, 
   CALLBACKS.NON-EXISTANT, CALLBACKS.UNINTERNED, MEM-AREF.BARE-STRUCT, 
   DEREF.ARRAY-OF-AGGREGATES, MAKE-POINTER.HIGH, STRUCT.ALIGNMENT.2, 
   STRUCT.ALIGNMENT.3, STRUCT.ALIGNMENT.4, STRUCT.ALIGNMENT.5, 
   STRUCT.ALIGNMENT.6, STRUCT.ALIGNMENT.8, 
   STRUCT-VALUES.TRANSLATION.MEM-AREF.1, SET-AGGREGATE-STRUCT-SLOT, 
   SET-AGGREGATE-STRUCT-SLOT.ACC, FSBV.WFO, FSBV.MAKEPAIR.1, 
   FSBV.MAKEPAIR.2, MISC-TYPES.BOOLEAN.1, 
   MISC-TYPES.BOOL.CONVERT-TO-FOREIGN.CALL, 
   MISC-TYPES.BOOL.CONVERT-FROM-FOREIGN.CALL.
26 unexpected failures: FUNCALL.CHAR, DEFCFUN.CHAR, DEFCFUN.NOOP, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.CHAR, CALLBACKS.LONG, CALLBACKS.BFF.1, CALLBACKS.BFF.2, 
   CALLBACKS.NON-EXISTANT, CALLBACKS.UNINTERNED, MEM-AREF.BARE-STRUCT, 
   DEREF.ARRAY-OF-AGGREGATES, MAKE-POINTER.HIGH, STRUCT.ALIGNMENT.2, 
   STRUCT.ALIGNMENT.3, STRUCT.ALIGNMENT.4, STRUCT.ALIGNMENT.5, 
   STRUCT.ALIGNMENT.6, STRUCT.ALIGNMENT.8, 
   STRUCT-VALUES.TRANSLATION.MEM-AREF.1, SET-AGGREGATE-STRUCT-SLOT, 
   SET-AGGREGATE-STRUCT-SLOT.ACC, MISC-TYPES.BOOLEAN.1, 
   MISC-TYPES.BOOL.CONVERT-TO-FOREIGN.CALL, 
   MISC-TYPES.BOOL.CONVERT-FROM-FOREIGN.CALL.

--> second execution (perhaps wrong)
and hang in FSBV.1

[kpoeck]

Second evaluation comes from:

(defun run-all-cffi-tests ()
  (append (run-cffi-tests :compiled nil)
          (run-cffi-tests :compiled t)))

[selwynsimsek]

(cffi-tests::run-all-cffi-tests)
->
.... 
19 out of 323 total tests failed: FUNCALL.CHAR, DEFCFUN.CHAR, DEFCFUN.NOOP, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.CHAR, CALLBACKS.BFF.1, CALLBACKS.BFF.2, 
   CALLBACKS.NON-EXISTANT, CALLBACKS.UNINTERNED, MAKE-POINTER.HIGH, 
   FSBV.WFO, FSBV.MAKEPAIR.1, FSBV.MAKEPAIR.2, MISC-TYPES.BOOLEAN.1, 
   MISC-TYPES.BOOL.CONVERT-TO-FOREIGN.CALL, 
   MISC-TYPES.BOOL.CONVERT-FROM-FOREIGN.CALL, SHAREABLE-VECTOR.1, 
   SHAREABLE-VECTOR.2.
16 unexpected failures: FUNCALL.CHAR, DEFCFUN.CHAR, DEFCFUN.NOOP, DEFCFUN.BFF.1, DEFCFUN.BFF.2, 
   CALLBACKS.CHAR, CALLBACKS.BFF.1, CALLBACKS.BFF.2, 
   CALLBACKS.NON-EXISTANT, CALLBACKS.UNINTERNED, MAKE-POINTER.HIGH, 
   MISC-TYPES.BOOLEAN.1, MISC-TYPES.BOOL.CONVERT-TO-FOREIGN.CALL, 
   MISC-TYPES.BOOL.CONVERT-FROM-FOREIGN.CALL, SHAREABLE-VECTOR.1, 
   SHAREABLE-VECTOR.2.

;; second execution hangs after  FSBV.WFO 

[ruffianeo]

I would love giving clasp a go and use it next to sbcl as my main common lisp implementation. But many of my must-have quicklisp libraries are not working in clasp - and all probably mainly due to CFFI problems.

(ql:quickload :cffi) ;; works
;; but libraries not loading include:
(ql:quickload :sdl2) ;; my graphics weapon of choice
(ql:quickload :gsll) ;;  Undefined reader macro for char '#' subchar '+' in file polynomial.lisp line: (60) column (2).
;; loading but not working:
(ql:quickload :vgplot) ;; for quick and dirty graphing - does not show window and so it is a glorified null device :)
(ql:quickload :ltk) ;; sends its output to *standard-output* instead of to tk...

... etc. etc. Anything which seems to rely on FFI does not seem to work.

Running all that on debian bullseye btw. With emacs/slime etc.

[kpoeck]

I know that this does not help, but the gsll error has nothing to do with ffi, it is because of (EDITOR-HINTS.NAMED-READTABLES:IN-READTABLE :ANTIK) not properly working.

[kpoeck]

Concretely since (get-dispatch-macro-character #\# #\+ (NAMED-READTABLES:find-readtable :antik)) returns nil, which it shouldn't

[kpoeck]

this is because of antik-master-df14cb8c-git/input-output/readtable.lisp the definition of antik should be

(named-readtables:defreadtable :antik
  (:merge :standard)
  #-(or ccl clasp)
  (:macro-char #\# :dispatch))

[Bike]

Should clasp behave differently on (:macro-char #\# :dispatch)? I haven't really used named readtables myself and don't know much about how it works.

[kpoeck]

(:macro-char #\# :dispatch) -> (make-dispatch-macro-character #\# nil <readtable>)

that calls:

T_sp Readtable_O::make_dispatch_macro_character_(Character_sp ch, T_sp non_terminating_p) {
  _OF();
  this->set_macro_character_(ch, _sym_dispatch_macro_character, non_terminating_p);
  this->DispatchMacroCharacters_->setf_gethash(ch, HashTableEql_O::create_default());
  return _lisp->_true();
}

that seems to reset all functions for all subchars, but doesn't in sbcl

so in clasp

COMMON-LISP-USER> (let ((new (copy-readtable)))
  (values
   (get-dispatch-macro-character #\# #\+ new)
   (make-dispatch-macro-character #\# nil new)
   (get-dispatch-macro-character #\# #\+ new)))

ECLECTOR.READER::SHARPSIGN-PLUS
T
NIL

sbcl

(let ((new (copy-readtable)))
  (values
   (get-dispatch-macro-character #\# #\+ new)
   (make-dispatch-macro-character #\# nil new)
   (get-dispatch-macro-character #\# #\+ new)))
#<FUNCTION SB-IMPL::SHARP-PLUS-MINUS>
T
#<FUNCTION SB-IMPL::SHARP-PLUS-MINUS>

what clasp does,might be reasonable, what do you think?

[kpoeck]

for whats it worth, ecl and ccl and clisp all do what clasp does and return nil for the last form

[ruffianeo]

I was aware while writing my above post, that the gssl problems are different from the others. This thread was the trigger to simply test all my (well not favorite but used before) packages with CLASP and while writing the above post, I was just adding the problems, I found.

@kpoeck One could argue, that a quicklisp package which is supposedly compatible across CL implementations SHOULD work on a new CL implementation as well. Even if the new CL makes some "better or equally good choice" on some design detail. The mere fact, that the package won't work, renders that choice as worse.

In the aftermath of my CLASP experiments, I gave CLISP the same treatment and it did not fare too well in terms of quicklisp package support, either. I guess, there is a reason, why SBCL is probably the most popular (non-commercial) implementation. So, it might be a good idea to benchmark "package support" of CLASP against SBCLs package support.

[Bike]

we generally try to prioritize standard conformance over just making libraries work. i could muster up a philosophical argument, like how one defined thing is more sensible to deal with than the decisions of dozens of uncoordinated authors, but that's the fact of the matter. so yes, "a quicklisp package which is supposedly compatible across CL implementations SHOULD work on a new CL implementation as well", and of course we would like for common lisp libraries to be usable, but sometimes that can mean an issue with the library rather than with clasp.

in this case the standard seems fairly clear:

Initially, every character in the dispatch table associated with the char [that make-dispatch-macro-character was called on] has an associated function that signals an error of type reader-error.

I suppose you could argue that "initially" means that this only applies the first time it's called, though.

...but with antik specifically, I don't actually understand what the intent is. The antik readtable merges the standard readtable, so # should already be a dispatching macro character, and if SBCL's make-dispatch-macro-character does not clear the dispatch table, doesn't designating # in the defreadtable there do nothing? The code doesn't seem to have been touched in several years, too. I would be curious to know what is supposed to happen.

as for the other issues, our FFI certainly has extant problems, such as #791. Unfortunately, we have only two real full time developers (one, when chris is off doing chemistry), and there's only so much we can do. FFI hasn't really been a priority for us at the moment as we use a different system to integrate with C++ code, and we primarily use jupyter as a GUI, which has pretty different interoperability issues. More detailed bug reports on the problems you have run into, backtraces, etc could be helpful.