Beneath the SPAD code and the Boot code is a Lisp interpreter. This provides much of the underling functionality such as Input/output, garbage collector and so on.
I think that relying on this lisp code, just under the surface, causes a number of problems:
- I would really like more mainstream base language that supports parallel code and more powerful libraries for graphics and so on. I think parallel code is very important, I think mathematics has a lot of inherent parallelism (vectors for example).
- The main issue is that I find it extremely difficult to work out what a given piece of Lisp/boot code does. If I look at a given function there is no clues about the structure of its parameters or what variables are being used or what they are used for. I think it's because Lisp/boot uses a single type (feels like dynamic typing) and global variables it is very hard to understand code.
- Lisp shows through underneath SPAD. That is debugging, error messages, outputs, etc. are often formatted as Lisp or require Lisp knowledge. I think this is particularly important for potential new users. SPAD has a steep enough learning curve on its own without needing to learn Lisp
- Lots of complications in supporting multiple flavors of Lisp (SBCL, clisp, etc.)
- Lots of complications in supporting multiple platforms (Windows, Android, etc.)
I would like to find a way to replace this lisp code with something else that does not have these problems (see this page for attempt to do this).
I can see that Lisp is quite mathematical (lambda calculus) and I can see that it allows the original programmer to be very creative, but when trying to understand other peoples code, creative is not what I need. I want a language with meaningful types and minimal global variables.
Options
Ideally it would be nice to swap out this low level lisp environment for something else but leave the high level SPAD as it is. I can't see an easy way to do this, the candidates I can think of are:
| Low Level Environment | Notes |
|---|---|
| Lisp | current situation for Axiom/FriCAS |
| FOAM | Custom environment for Aldor |
| JVM | used for Java |
| LLVM |
But these options all seem to have problems:
Lisp
This is current situation, see issues above.
FOAM
This is a custom environment for Aldor so would require resources to keep it maintained. I doubt it would support parallel code.
JVM
I like the Java environment, there are lots of IDEs and library code available, I think good graphics libraries would be an enormous benefit for Axiom/FriCAS. However, most people on the FriCAS forum hate java so I don't think there would be much support for that.
JVM does not support dependant types directly, but I don't think that's a problem? because SPAD would be built on top of this environment, not in it.
LLVM
LLVM itself does not provide a garbage collector, you must provide your own, there are hooks provided and external packages available. We would need accurate (as opposed to conservative) garbage collection which supports parallel code.
Accurate garbage collection requires the ability to identify all pointers in the program at run-time (which requires that the source-language be type-safe in most cases). Identifying pointers at run-time requires compiler support to locate all places that hold live pointer variables at run-time, including the processor stack and registers.
I don't know much about this topic but (in my ignorance) this looks to me like a horrible thing to implement. In my nightmares I can imagine months spent trying to get obscure code from different sources to work together and track down nasty memory leaks.
More Short Term Options
Since the above options all look impossibly hard for one person (and there is not much interest in joint projects) I decided to look for something simpler that might be a small step along the way.
I have been looking at converting boot code to SPAD (described here).
I managed to compile this boot code to an Abstract Syntax Tree (AST) I then wrote a code generator to write AST back to boot code. This code was different from the original (because the AST does not store format information) but when I compiled to Lisp the code generated was identical - all lines (except those starting with ';') were identical.
So I am sure my AST contains all the information from the boot files.
Its then not difficult to generate SPAD or Aldor or any other language from this AST. But the problems are:
- how to handle global variables, especially dynamic variables. I think each function would have to have an additional parameter to pass around the runtime values of these dynamic variables.
- The boot 'where' keyword seems to hold inner functions and variables. How should inner functions be converted to SPAD? SPAD has lambdas but can SPAD lambda capture variables (closures)?
- The native Lisp code would have to be translated by hand, to handle input/output to console, files and database (which would be different for each language).
This file compiles the boot code to an AST.
- In order to get the compiler to compile I had to make a few tweaks to the boot code, these files. compile properly.
- This file generates boot code from the AST.
- Other files allow the compiler to distinguish blocks (pile mode).
- Since I now have confidence in the AST I can now go on to generate SPAD from the AST. I have started this in this file but this is work in progress and I still have a lot more to do on this.
Perhaps another option would be to just translate boot to SPAD manually but that's hard because Boot/Lisp passes everything around in global variables whereas well written SPAD should encapsulate information in domains. This means you cant just translate one function at a time and check if it works before going on to the next.
I think it would be hard to change one function at a time because all these functions are linked by lots of global variables.
I suspect it would only be possible to untangle all that mess if you have a very deep understanding of the interpreter and of boot. Since I would like to remove boot code I would rather not spend a lot of time learning about it.
One thing that I can easily do, since I have an AST, is generate tables showing which functions use which variables and another showing which variables are used in which functions. But I think it would take more knowledge than I have to untangle them. I might even be able to infer which variables only get assigned to one type (int,float,string..) so that they could be assigned those types in SPAD. However many boot variables could be polymorphic or complicated list structures.
But how do you generate static type language from a dynamic type language?
I mapped functions in boot to functions in SPAD. For convenience I just put them in packages one for each boot file.
Variables in boot can map to variables which are instances of SExpression. This instance of SExpression can have have common lisp functions like '+' defined. At runtime this function would add if its a number and fail if its not. So its checking the type dynamically at runtime - just like boot/lisp.
Krystian Bacławski s Project
Higher Level Options
Other Issues
Discussion with Tim Daly on forum here,The issue of the current lisp code is related to several factors. Scratchpad was implemented in maclisp, ported to lispvm, and then ported to common lisp (I did large portions of the CL port). So the code is a macro-mangled version of several lisps. In addition, there was another language (called Meta) that was part of the build process making life more complex. Boot added to that complexity by making it unreasonable to write good lisp code that takes advantage of Common lisp. Nobody writes CL code using lists of lists of lists and cadadr accesses. Common lisp code can be very readable. It is easy to write in a functional style and structure the data in a more rational fashion. I've been working in that direction. You're right that nobody is going to change their minds about this issue, of course. It's one of the reasons for a fork. But if you're going to put forth the effort to rewrite the compiler in Spad it might be worthwhile to create a specification of what it does now before trying to write Spad code. There are a lot of subtle hacks (e.g. add chain searches) that would be useful to explain. Another thought is that Aldor was an attempt to re-write the world into a higher level language. I spent 4 years of PhD research trying to move Axiom's algebra library into Aldor. Aldor was supposed to be the replacement compiler for Scratchpad. Unfortunately it suffers from "the second system effect", losing sight of the primary goal. You could consider moving the algebra to Aldor but I don't recommend it. Another thought... While people don't seem to like lisp as a platform it does have major advantages. I usually find myself using the lisp debugging facilities (trace, break, and REPL evaluation) to find out what failed and where. If you go from Spad to executable code you lose all of that. |
Tim, This is one of the problems, in my view, in that Lisp shows through underneath SPAD. That is debugging, error messages, outputs, etc. are often formatted as Lisp or require Lisp knowledge. I think this is particularly important for potential new users. SPAD has a steep enough learning curve on its own without needing to learn Lisp. Could it be things like this that prevent new user take up and explain the limited amount of discussion here? What I would like would be a clean language for people interested in mathematics without all the baggage of having to learn lots of (non-mainstream) computer stuff that's not really about the mathematics. Martin B |
That seems reasonable, except for the fact that any non-algebra error that does show up has to exhibit itself somehow. Worst case is a core dump. Spad does not define robust error handling primitives. Local try-catch is reasonable but it is not clear how to handle things like restarts, tracebacks, etc. when the non-algebra error occurs. How would you model handling errors in Spad? I do think that there might be an interesting research question of how to handle classes of errors in computational mathematics. I had proposed using Provisos to handle side-conditions on formulas. Hoon Hong and Chris Brown have done a lot of work on QEPCAD for handling these. Manuel Bronstein and I had long discussions about a SUCHTHAT domain for encapsulating Provisos but little code resulted as the QEPCAD work was still in the future at the time. As for writing Axiom in Spad, you find that compiling Axiom to efficient but generic code relies heavily on trampolines and symbol-plist lookups to do dynamics dispatch. I can't imagine how to work that into a strong type system so a compiler written in Spad might be a challenge. Grab a copy of Axiom and try to move some of the code to Spad. I'd recommend starting with eliminating any $Lisp calls in the algebra. That will move the algebra away from the underlying Lisp. It will also bring a lot of issues into focus. A non-lisp-based Spad cannot have $Lisp calls so each would have to be replaced with a Spad implementation. On the other hand, my current effort involves proving Axiom correct. That should (in theory) eliminate whole classes of errors. This is at the expense of proving new code correct which tends to get a negative reaction from developers. The upside is that "underneath Spad errors" should not occur. Proven code certainly moves away from "lots of (non-mainstream) stuff that's not really about the mathematics". On the other hand, it does make the mathematics hill-climb steeper. Unless you're up to speed on Heyting and Intuitionistic Logic, writing proven code is going to ruin your weekends, based on my current weekend experience :-) Axiom is fundamentally a research platform in the area of computational mathematics. I expect the user base to remain rather small, especially as it gets closer to a rigorous mathematical base. Tim |