Introduction
The Lux programming language is a functional language belonging to the Lisp family.
It features a flexible and expressive static type-system, and it's meant to run in a variety of different platforms.
Lux is committed to the functional style of program design, being a purely-functional programming language, while also adopting eager-evaluation over lazy-evaluation, to promote simpler reasoning over the performance and behavior of programs.
Lux also offers novel features in the area of meta-programming, with first-class types that can be examined and constructed at compile-time, monadic macros with access to the state of the compiler, and a style of macro definition that promotes composition and easy interaction between different macros.
While the richness and variety of what Lux has got to offer is much larger than what can be described in this introduction, hopefully I've already mentioned enough to stimulate the curiosity of those interested in advanced concepts in programming languages and computer science, and those engineers seeking powerful tools for both program design and implementation.
Lux is both a simple and a complex language.
Its design allows you to make effective programs with just a small subset of what it has to offer, but the goal of the language is to provide its users with an arsenal of powerful tools to suit their various needs in their projects.
Finally, I must note that Lux is a practical language, meant for day-to-day usage by software engineers, instead of just research and experimentation by academics.
It may seem unnecessary to point that out, but both Lisp-like languages and functional languages have earned a reputation for being academic in nature.
While Lux's design does involve a lot of advanced ideas in computer science, it is with the intention of turning Lux into a powerful and effective tool for day-to-day software engineering.
It is my hope that within these pages the reader will find both a host of new ideas to enrich their perspective on programming, and the promise of great power should they choose to add Lux to their arsenal of programming languages.
I wish you, my dear reader, good luck on this journey, and much fun!
Click here to read the 1st chapter.
Copyright Note
This work is under the Mozilla Public License 2.0, and was published as part of the the Lux repository on GitHub.
Chapter 1: Getting started
Where you will learn how to set up a development environment for Lux.
Before any coding can happen, it is necessary to set-up everything you need to become a productive Lux programmer.
Question #1: How do I write Lux code?
Text editor support is a fundamental thing for any language, and Lux already covers some of that. The catch is that there's only support for Emacs at the moment.
The plugin is called lux-mode.
The instructions for how to install it are at the link and it won't take much time.
Note: If you've already installed lux-mode before while using a previous version of Lux, you should install it again, as the language has changed a lot between previous versions and v0.6.
Question #2: How do I build Lux programs?
Lux uses a custom-made build tool named Aedifex which is configured using a declarative Lux-based syntax.
To install Aedifex, go to https://github.com/LuxLang/lux/tree/master/shell and download either lux.bat or lux.sh depending on whether you're on Windows or Linux/Mac.
Also download the aedifex.jar file, and place it (along with either of the scripts you downloaded) somewhere in your PATH.
Now, you'll have access to the lux command, which allows you to run Aedifex to build and test Lux projects.
Question #3: How do I use Aedifex?
To find out, let's create a sample project that will have everything we need.
These are the steps:
- Create a directory called
my_project. - Create a new project file at
my_project/project.lux. - Add this to the project file:
[""
["identity" ["my.group" "my_project" "0.1.0-SNAPSHOT"]
"repositories" ["https://oss.sonatype.org/content/repositories/snapshots/"
"https://oss.sonatype.org/service/local/staging/deploy/maven2/"]
"dependencies" [["com.github.luxlang" "stdlib" "0.7.0" "tar"]]
"lux" ["com.github.luxlang" "lux-jvm" "0.7.0" "jar"]
"program" main.main]]
... By default, Aedifex uses the "source" directory for finding your source-code.
... The file containing our program will be my_project/source/main.lux.
- Create
my_project/source/main.luxand add this code to it:
(.require
[library
[lux (.except)
[program (.only program)]
["[0]" debug]
[control
["[0]" io]]]])
(def main
(program args
(io.io (debug.log! "Hello, world!"))))
... As you can see, this is nothing more than a very simple "Hello, world!" program to test things out.
... Everything will be explained later in the rest of the book.
- In your terminal, go to
my_project, and executelux build.
When it's done, you should see a message like this:
...
Compilation complete!
Duration: +15s26ms
[BUILD ENDED]
A directory named target will have been created, containing everything that was compiled, alongside an executable JAR file.
-
Run the program with this command:
java -jar target/program.jar -
Smile :)
For a thorough specification of what Aedifex can do, please refer to Appendix H.
Question #4: Where can I find documentation for Lux?
A specially useful source of information is the documentation for the standard library.
You can also explore the Lux repository on GitHub for more information.
Question #5: Where do I talk about Lux?
The place to talk about Lux is at the Lux forum.
You can also come and say hi at the Gitter channel.
And, to communicate directly with me, just email me at: luxlisp@gmail.com
Now, we can proceed to the actual teaching of the language!
See you in the next chapter!
Chapter 2: The basics
Where you will learn the fundamentals of Lux programming.
Modules
Lux programs are made of modules.
A module is a file containing Lux code, and bearing the extension .lux at the end of its name (like our main.lux file).
Modules contain a single module statement, various definitions and a few other kinds of statements as top-level code (that is, code that is not nested within other code).
Definitions
Definitions are the top-level or global values that are declared within a module.
They may be of different types, such as constant values or functions, or even fancier things like types, interfaces or implementations (more on those in later chapters).
Also, definitions may be private to a module, or exported so other modules can refer to them.
By default, all definitions are private.
Values
Values are just entities which carry some sort of information.
Every value has a type, which describes its properties.
Lux supports a variety of basic and composite values:
- Bit:
#0and#1values. They take the role of boolean values in other languages, and thefalseandtrueconstants mask them for greater familiarity. - Nat: Unsigned 64-bit integers.
- Int: Signed 64-bit integers.
- Rev: Unsigned 64-bit numbers in the interval [0,1).
- Frac: Signed 64-bit floating-point numbers.
- Text: Strings.
- Unit: A special value that sort-of represents an empty value or a non-value.
- Variant: A value of a particular type, from a set of heterogeneous options.
- Tuple: An ordered group of heterogeneous values which may be handled as a single entity.
- Function: A first-class function or procedure which may be invoked, or passed around like other values.
Types
Types are descriptions of values that the compiler uses to make sure that programs are correct, and invalid operations (such as multiplying two texts) are never performed.
The thing that makes Lux types special is that they are first-class values, the same as bits and ints (albeit, a little more complex).
They are data-structures, and they even have a type... named Type (I know, it's so meta).
We'll talk more about types in later chapters.
Macros
Macros are special functions that get invoked at compile time, and that have access to the full state of the compiler.
The reason they run during compilation is that they can perform transformations on code, which is a very useful thing to implement various features, DSLs (domain-specific languages) and optimizations.
We'll also explore macros further in later chapters.
Comments
... They look like this.
... They all start with 3 contiguous . characters and go on until the end of the line.
Expressions
An expression is code that may perform calculations in order to generate a value.
Data literals (like int, tuple or function literals) are expressions, but so are function calls, pattern-matching and other complex code which yields values.
Macro calls can also be involved if the macro in question generates code that constitutes an expression.
Directives
Directives looks similar to expressions, except that their purpose is not to produce a value, but to communicate something to the compiler.
This is a bit of a fuzzy line, since some things which also communicate stuff to the compiler are actually expressions (for example, type annotations, which we'll see in next chapter).
Examples of directives are .require declarations at the top of modules, and definitions of all kinds (such as program definitions).
Programs
Lux doesn't have special "main" functions/procedures/methods that you define, but the program macro accomplishes the same thing and works similarly.
It takes a list of command-line inputs and must produce some sort of action to be performed as the program's behavior.
That action must be of type (IO Any), which just means it is a synchronous process which produces any value (regardless of type) once it is finished.
Command-Line Interface
Lux programs can have graphical user interfaces, and in the future they may run in various environments with much different means of interfacing with users, or other programs.
But as a bare minimum, the Lux standard library provides the means to implement command-line interfaces, through the functionality in the parser/lux/program module.
That module implements a variety of parsers for implementing rich command-line argument processing, and you should definitely take a look at it once you're ready to write your first serious Lux program.
Functional Programming
This is the main paradigm behind Lux, and there are a few concepts that stem from it which you should be familiar with:
- Immutable Values: The idea is that once you have created a value of any type, it's frozen forever. Any changes you wish to introduce must be done by creating a new value with the differences you want. Think, for instance, of the number 5. If you have 2 variables with the same number, and you decide to change the value in one variable to 8, you wouldn't want the other variable to be affected. Well, the same idea applies to all values. This is clearly a departure from the imperative and object-oriented style of having all data be mutable, but it introduces a level of safety and reliability in functional programs that is missing in the imperative style.
- First-Class Functions: This just means that functions are values like any other. In most languages, functions/methods/procedures are more like features you register in the compiler for later use, but that just remain static in the background until you invoke them. In functional programming, you can actually pass functions as arguments to other functions, or return them as well. You can store functions in variables and inside data-structures, and you can even produce new functions on the fly at run-time.
- Closures: Functions that get generated at run-time can also "capture" their environment (the set of local variables within the function's reach), and become closures. This is the name for a function which "closes over" its environment, making it capable to access those values long after the function was originally created. This allows you to create functions as templates which get customized at run-time with values from their environment.
Now, let's talk a bit more about the program we saw last time.
In the previous chapter we compiled and ran a Lux program, but nothing has been explained yet. Let's review the code and see in detail what was done.
... This will be our program's main module.
(.require
[library
[lux (.except)
[program (.only program)]
["[0]" debug]
[control
["[0]" io]]]])
(def main
(program args
(io.io (debug.log! "Hello, world!"))))
The first part of this program specifies which dependencies we require.
All Lux modules automatically import the library/lux module, but they don't locally import every single definition, so everything would have to be accessed by using the library/lux. prefix or the . (short-cut) prefix.
To avoid that, we import the library/lux module in a plain way.
By the way, what I just explained about the `library/lux` module is the reason why we couldn't just use the `.require` macro as `require`.
Then we import the library/lux/control/io module. We're giving this module an alias, using that "[0]" syntax. The way aliasing works here is that it replaces the [0] with the short name of the import, and so [0] becomes io, and that is the alias given to the import. The same process happens when we import the library/lux/debug module. This might seems weird and sort of useless, but the aliasing syntax has some more features and flexibility, enabling you to have your own naming convention when importing modules.
Notice how we express nested modules (up to arbitrary depths) by simply nesting in brackets.
Finally, we import the library/lux/program module. Notice how the syntax is a bit different in this case. Here, we're saying that we don't want to locally import any definition within it, only program. We use a similar syntax when importing the library/lux module. That just means we want to locally import all the public definitions, except none.
Now, let's analyse the actual code!
We're defining the entry point of our program (what in many other languages is referred to as the main function/procedure/method). We'll be receiving all the command-line arguments in a (List Text) called args, and we must produce a value of type (IO Any) (the IO type being defined in library/lux/control/io, but omitted here for brevity).
We'll go into more detail about what `IO` and `Any` mean in the next chapter.
Suffice it to say that the debug.log! function will produce a value of type Any after printing/logging our "Hello, world!" text, and the io.io macro will wrap that in the IO type.
That (IO Any) value will then be run by the system at run-time, giving us the result we want.
Now that we've discussed some of the basics of what goes on inside of Lux programs, it's time for us to explore the language in a little bit more depth.
See you in the next chapter!
Chapter 3: Syntax and data-types
Where you will learn the what Lux code is made of.
Syntax for data-types
Bits look like this:
#0 ... false
#1 ... true
Nats look like this:
0
123
0456
123,456,789
Ints look like this:
+0
-123
+0456
-123,456,789
Revs look like this:
.123
.04,56
.7890
Fracs look like this:
+123.456
-456.7890
+0.001
123,456.789
Texts look like this:
"This is a single-line text"
Unitlooks like this:
[]
- Variants look like this:
{#Foo}
{#Bar 10 +20.0 "thirty"}
- Tuples look like this:
[123 ["nested" #tuple] true]
- Records look like this:
[#name "Lux" #paradigm {#Functional} #platforms (list {#JVM})]
Note: As you can see, commas (,) can be used as separators for the numeric literals.
Regarding those label thingies we saw earlier, they're called labels, and the reason they're not mentioned as part of Lux's data-types is that they're not really data-types; they're just part of the language syntax.
They're used as part of the syntax for data-types, but they're not data-types in themselves.
Also, you can't just use anything you want as a label, as you first have to declare them.
We'll talk more about labels a bit later, when we talk about defining types.
Also, just from looking at the syntax for unit and tuples, you can see that they look quite similar. The reason is that unit is actually the empty tuple. I know it sounds odd, but for the most part you just have to think of unit as a kind of empty value, considering that it contains no information inside.
It might sound specially odd that we have an "empty" value at all in the first place, but as it turns out, it's quite useful in a variety of situations.
You're about to see one of those pretty soon.
In the section for variants, you can see 2 different alternatives, and you might wonder how do they differ.
Well, a variant is a pair of a label (referred to as a tag) and a single value.
That's right, I said single value; so you might be wondering how come we're associating 3 values with the #Bar tag.
It's pretty simple, actually: whenever you're trying to create a variant with more than one value, Lux just wraps all the values inside a tuple for you.
So, {#Bar 10 +20.0 "thirty"} is the same as {#Bar [10 +20.0 "thirty"]}.
Now, you might be thinking: what's up with that #Foo variant?
Well, sometimes you only care about a variant for its tag, and not for any value it may hold (for example, if you're trying to use a variant type as an enumeration).
In that case, you'll want to pair the tag with an empty value (since it has to be paired with something).
That's right! You've just witnessed unit value in action and you didn't even know it.
If you just write the tag in braces, with nothing else in there, Lux will stick a unit in there for you.
That means {#Foo} is the same as {#Foo []}.
You might have noticed that I mentioned records in this chapter, but not in the previous chapter, where I also talked about the basic data-types Lux offers.
The reason is that records are a bit of a magic trick in Lux.
That means records are not really a data-type that's distinct from the other ones.
In fact, records just offer you an alternative syntax for writing tuples.
That's right! [#name "Lux" #paradigm {#Functional} #platforms (list {#JVM})] could mean the same as ["Lux" {#Functional} (list {#JVM})], depending on the ordering imposed by the slots (which is the name given to record labels).
Remember when I said that you needed to declare your labels?
Well, depending on the order in which you declare them, that means that #name could point to the first element in the tuple, or to another position entirely.
Also, in the same way that labels have a numeric value when it comes to their usage in tuples/records, that's also the case for variants.
For example, the List type has two tags: .#End and .#Item.
The .#End tag has value 0, while the .#Item tag has value 1.
That's what allows Lux to the able to identify which option it is working with at runtime when you're dealing with variants.
Labels belong to the module in which they were declared, and you must use the module name (or an alias) as a prefix when using tags.
That is why I've written .#End and .#Item, instead of #End and #Item.
However, you may forgo the prefixes if you're referring to tags which were defined in the same module in which they're being used.
Types for data-types
"But, wait!", you might say. "We didn't talk about functions!"
Patience, young grasshopper. We'll talk about those in the next chapter.
For now, let's talk about types.
The type-annotation macro is called is. You use it like this: (is Some_Type some_value).
There is also a separate macro for type-coerciones that's called as, which is used the same way. However, you should probably steer clear off that one, unless you know what you're doing, since you can trick the compiler into thinking a value belongs to any type you want by using it.
Now that we know about type annotations, I'll show you some types by giving you some valid Lux expressions:
(is Bit #1)
(is Bit .true)
(is Nat 123)
(is Int -123)
(is Rev .789)
(is Frac +456.789)
(is Text "YOLO")
(type Some_Enum
(Variant
{#Primitive}
{#Variant}
{#Tuple}))
(is [Int [Text Some_Enum] Bit]
[10 ["nested" {#Tuple}] .false])
(type Quux
(Variant
{#Foo}
{#Bar Int Frac Text}))
(is Quux {#Foo})
(is Quux {#Bar 10 +20.0 "thirty"})
(type Lang
(Record
[#name Text
#paradigm Paradigm
#platforms (List Platform)]))
(is Lang
[#name "Lux"
#paradigm {#Functional}
#platforms (list {#JVM})])
(is Lang
["Lux" {#Functional} (list {#JVM})])
(is [Text Paradigm (List Platform)]
[#name "Lux"
#paradigm {#Functional}
#platforms (list {#JVM})])
By the way, the value of a type-annotation or a type-coearcion expression is just the value being annotated/coerced. So `(is Bit #1)` simply yields `#1`.
What is that type thingie?
It's just a macro for defining types. We'll learn more about it in a future chapter.
The labels that get mentioned in the type definition get automatically declared, and the order in which they appear determines their value.
#Foo came first, so its value is 0. #Bar, as you may guess, gets the value 1.
Also, you might be wondering what's the difference between List and list. Well, the first one is the type of lists (or a type-constructor for list types, however you want to look at it). The second one is a macro for constructing actual list values. List can only take one argument (the type of the list elements), while list can take any number of arguments (the elements that make up the list value).
Again, we haven't mentioned functions. But the next chapter is about them.
See you in the next chapter!
Chapter 4: Functions and definitions
Where you will learn how to build your own Lux code.
OK, so you've seen several explanations and details so far, but you haven't really seen how to make use of all of this information.
No worries. You're about to find out!
First, let's talk about how to make your own functions.
(function (plus_two x) (++ (++ x)))
Here's the first example.
This humble function increases a Nat twice.
What is its type?
Well, I'm glad you asked.
(is (-> Nat Nat)
(function (plus_two x) (++ (++ x))))
That -> thingie you see there is a macro for generating function types.
It works like this:
(-> arg1 arg2 ,,, argN return)
The types of the arguments and the return type can be any type you want (even other function types, but more on that later).
How do we use our function? Just put it at the beginning for a form:
((function (plus_two x) (++ (++ x))) 5)
... => 7
Cool, but... inconvenient.
It would be awful to have to use functions that way.
How do we use the plus_two function without having to inline its definition (like the debug.log! function we used previously)?
Well, we just need to define it!
(def plus_two
(is (-> Nat Nat)
(function (_ x)
(++ (++ x)))))
Or, alternatively:
(def plus_two
(-> Nat Nat)
(function (_ x)
(++ (++ x))))
Notice how the def macro can take the type of its value before the value itself, so we don't need to wrap it in the type-annotation : macro.
Now, we can use the square function more conveniently.
(plus_two 7)
... => 9
Nice!
Also, I forgot to mention another form of the def macro which is even more convenient:
(def (plus_two x)
(-> Nat Nat)
(++ (++ x)))
The def macro is very versatile, and it allows us to define constants and functions.
If you omit the type, the compiler will try to infer it for you, and you will get an error if there are any ambiguities.
You will also get an error if you add the types but there's something funny with your code and things don't match up.
Error messages keep improving on each release, but in general you'll be getting the file, line and column on which an error occurs, and, if it's a type-checking error, you'll usually get the type that was expected and the actual type of the offending expression... in multiple levels, as the type-checker analyses things in several steps.
That way, you can figure out what's going on by seeing the more localized error alongside the more general, larger-scope error.
Functions, of course, can take more than one argument, and you can even refer to a function within its own body (also known as recursion).
Check this one out:
(def (factorial' acc n)
(-> Nat Nat Nat)
(if (n.= 0 n)
acc
(factorial' (n.* n acc) (-- n))))
(def (factorial n)
(-> Nat Nat)
(factorial' 1 n))
And if we just had the function expression itself, it would look like this:
(function (factorial' acc n)
(if (n.= 0 n)
acc
(factorial' (n.* n acc) (-- n))))
Here, we're defining the factorial function by counting down on the input and multiplying some accumulated value on each step.
We're using an intermediary function factorial' to have access to an accumulator for keeping the in-transit output value, and we're using an if expression (one of the many macros in the library/lux module) coupled with a recursive call to iterate until the input is 0 and we can just return the accumulated value.
As it is (hopefully) easy to see, the if expression takes a test expression as its first argument, a "then" expression as its second argument, and an "else" expression as its third argument.
Both the n.= and the n.* functions operate on Nats, and -- is a function for decreasing Nats; that is, to subtract 1 from the Nat.
You might be wondering what's up with that n. prefix.
The reason it exists is that Lux's arithmetic functions are not polymorphic on the numeric types, and so there are similar functions for each type.
If you import the module for Nat numbers, like so:
(.require
[library
[lux
[math
[number
["n" nat]]]]])
You can access all definitions in the library/lux/math/number/nat module by just using the n. prefix.
Also, it might be good to explain that Lux functions can be partially applied.
This means that if a function takes N arguments, and you give it M arguments, where M < N, then instead of getting a compilation error, you'll just get a new function that takes the remaining arguments and then runs as expected.
That means, our factorial function could have been implemented like this:
(def factorial
(-> Nat Nat)
(factorial' 1))
Or, to make it shorter:
(def factorial (factorial' 1))
Nice, huh?
We've seen how to make our own definitions, which are the fundamental components in Lux programs.
We've also seen how to make functions, which is how you make your programs do things.
Next, we'll make things more interesting, with branching, loops and pattern-matching!
See you in the next chapter!
Chapter 5: Control flow
Where you will learn how to give intelligence to your code.
Branching
So far, we've only seen very simple examples of program/function logic that are very straightforward.
But real world programs are far from straightforward, and there's a lot of testing and decision-making involved in how they operate.
The first important concept to master in any programming language is how to make decisions and branch the path our program takes, and Lux offers one primary way of doing this: pattern-matching.
But before we head into that, let's first see 2 weaker mechanisms for branching that might be familiar to programmers coming from other programming languages.
If
We've already met the humble if expression in the previous chapter.
As explained there, the expression takes the following form:
(if test
then
else)
Where test, then and else are arbitrary Lux expressions.
In terms of types, it works like this:
(is X (if (is Bit test)
(is X then)
(is X else)))
Here is an example:
(if true
"Oh, yeah!"
"Aw, hell naw!")
So, both branches must produce values of the same type for the type-checker to let it pass.
Cond
cond is like a more general version of the if macro.
For those of you coming from conventional programming languages, cond is like a chain of if-else statements/expressions, with a default else branch at the end, in case all fails.
It looks like this:
(cond test-1 then-1
test-2 then-2
...
test-n then-n
else)
And, in terms of types, it looks like this:
(is X (cond (is Bit test-1) (is X then-1)
(is Bit test-2) (is X then-2)
...
(is Bit test-n) (is X then-n)
(is X else)))
Here is an example:
(cond (n.even? num) "even"
(n.odd? num) "odd"
... else-branch
"???")
So, it's easy to intuit how cond would de-sugar into several nested if expressions.
Also, I'd like to point out that both if and cond are macros, instead of native Lux syntax.
The reason for that is simply that they can both be implemented in terms of pattern-matching.
Pattern-matching
Some of you may not be familiar with the concept of pattern-matching if you come from non-functional programming languages, or from FP languages that lack pattern-matching (e.g. Clojure).
Pattern-matching is similar to the branching expressions that we saw previously, except that instead of being based on making boolean tests, it's based on comparing patterns against data, and executing a branch if the pattern matches that data.
The beautiful thing is that the pattern can be very complicated (even involving the binding of variables), and so the testing can be very powerful.
We can see its power by looking at some examples.
For instance, the factorial' function you saw in the previous chapter could have been written like this:
(def (factorial' acc n)
(-> Nat Nat Nat)
(when n
0 acc
_ (factorial' (n.* n acc) (-- n))
))
As you may imagine, when is the pattern-matching macro.
It takes the data you want to pattern-match against (in this case, the n variable), and then tests it against several patterns until it finds a match, in which case it executes its branch.
Here, we test if n equals 0. If it does, we just return the acc value.
Otherwise, we have a default branch with a pattern that doesn't test anything called _. That will handle the case where the input is greater than 0.
The "default" branch works because we're binding the value of n onto a variable called _, and binding always succeeds, which is why we can use that branch as a default.
However, since it is binding a variable, that means we could have used _ instead of n during our calculations; like this:
(def (factorial' acc n)
(-> Nat Nat Nat)
(when n
0 acc
_ (factorial' (n.* _ acc) (-- _))
))
However, as a convention, _ is used as the name for values we don't care about and don't plan to use in our code.
Pattern-matching doesn't limit itself only to Nats, and can also be used with Bits, Ints, Revs, Fracs, Texts, variants, tuples, records, and much more!
Regarding the "much more" claim, you should check out Appendix C, where I discuss a powerful extension mechanism called pattern-matching macros.
Here are a couple more examples so you can see the possibilities.
(let [test true]
(when test
#1 "Oh, yeah!"
#0 "Aw, hell naw!"
))
(when (list 1 2 3)
{.#Item x {.#Item y {.#Item z {.#End}}}}
{.#Some (n.+ x (n.* y z))}
_
{.#None})
In the first example, you'll notice that we have rewritten the prior if example in terms of pattern-matching.
Also, you'll notice the introduction of a new macro, called let.
let is a simple way to create local-variables in Lux.
Its syntax looks like this:
(is X (let [var-1 expr-1
var-2 expr-2
...
var-n expr-n]
(is X body)))
Where the types of the variables will correspond to those of their matching expressions, and the type of the let expression will be the same as that of its body.
Also, remember when I told you that you can use pattern-matching to bind variables?
Well, guess what! let is implemented in terms of when, and it just gives you a more convenient way to bind variables than to go through all the trouble of doing pattern-matching.
Now, in the second example, we're deconstructing a list in order to extract its individual elements.
The List type is defined like this:
(type (List a)
{#End}
{#Item a (List a)})
.#End represents the empty list, while .#Item constructs a list by prepending an element to the beginning of another list.
With pattern-matching, we're opening our list up to 3 levels in order to extract its 3 items and do a simple math calculation.
If the match succeeds, we produce a value of type (Maybe Int) by wrapping our result with the .#Some tag, from the Maybe type.
If the match fails, we just produce nothing, by using the .#None tag, also from the Maybe type.
While List allows you to group an arbitrary number of values into a single structure, Maybe is for values you may or may not have.
Also, you may have noticed how different (and uncomfortable!) it is to pattern-match against a List, since you have to use its real syntax, with its tags; whereas to build the list we can just piggy-back on the list macro.
Don't worry too much about it, because there's a better way to do it that also allows us to use the list macro. If you're curious about it, head over to Appendix C to learn more about pattern-matching macros.
Looping
Alright. So, we know several ways to branch, and also how to bind variables.
But we know life isn't just about making decisions.
Sometimes, you just have to do your work over and over again until you're done.
That's what looping is for!
Recursion
In functional programming, recursion is the main mechanism for looping in your code.
Recursion is nothing more than the capacity for a function to call itself (often with different parameters than the initial call).
It's not hard to see how this mechanism can be used to loop in any way you want, and we've already seen examples of recursion in action.
(def (factorial' acc n)
(-> Nat Nat Nat)
(if (n.= 0 n)
acc
(factorial' (n.* n acc) (-- n))))
The factorial' function calls itself with an ever increasing accumulator (that will hold the eventual result), and an ever decreasing input value.
Recursion in many languages is seen as a risky operation, since programming languages have what are called "stacks", which are structures holding the parameters to functions and the return addresses for where to send the results once the functions are done.
Every function call you issue pushes a new frame onto the stack, and if enough frames are pushed, eventually the stack "overflows" its capacity, causing the program to fail.
However, an old trick that has been employed for a long time in programming languages is called tail-call optimization, and it allows you to optimize recursive calls that are in a "tail position"; that is, a position where the result of the call can just be returned immediately, instead of needing any further processing.
Our example factorial' function has its recursive call in the tail position (and thus, can benefit from that optimization).
This alternative doesn't:
(def (factorial' acc n)
(-> Nat Nat Nat)
(if (n.= 0 n)
acc
(n.+ 0 (factorial' (n.* n acc) (-- n)))))
Can you spot the difference?
In the alternative, the result of the recursive call would have to be sent to the n.+ function as its second parameter, so it wouldn't be a tail-call.
The beautiful thing about tail-call optimization (or TCO) is that it removes the recursion altogether, eliminating the possibility for stack overflows.
Pretty neat, huh?
For the sake of correctness, I must point out that Lux doesn't implement _full_ tail-call optimization, since that would require some extra power that Lux can't implement, since it's too low-level and (currently) the JVM doesn't offer the means to achieve that.
For that reason, going forward, I will refer to what Lux does as _tail-recursion optimization_ (or _TRO_), instead.
Loop
Some of you may be more familiar with the for loops or the while loops of other programming languages and need some time to wrap your heads around recursion. That's OK.
Lux also offers a macro that gives you a slightly similar experience to those kinds of loops, which can get your mind off recursion just a little bit.
To see it in action, let's rewrite (once more!) our factorial function:
(def (factorial n)
(-> Nat Nat)
(loop (again [acc 1
n n])
(if (n.= +0 n)
acc
(again (n.* n acc) (-- n)))))
We have eliminated our dependency on the factorial' function.
Just like with let, we're creating some local variables, but these are going to change on each iteration.
Then, in the body, we perform the usual if test, and if the number is not 0, then I use the again operator (which only works inside of loop) to update the values of my variables for the next iteration.
Piping
Piping isn't really control-flow per se, but I include it here because it is a powerful technique for organizing code that involves taking multiple steps to perform a calculation.
It's based on using a single macro, called |>, which allows you to write deeply nested code in a flat way, making it much easier to read and understand, and also giving it a bit of an imperative flavor.
Here is a simple example to see how it works:
(|> elems
(each to_text)
(interposed " ")
(mix append_text ""))
... =>
... (mix append_text ""
... (interposed " "
... (each to_text elems)))
If you read carefully, you'll see that each element (from left to right) gets lodged at the end of the next expression and the pattern continues until everything has been nested.
A good convention to follow in functional programming (and especially in Lux), is that the most important argument to a function (or its subject) ought to be the last argument the function takes.
One of the really cool benefits of this convention is that your code becomes very amenable to piping, as the nesting is only done in one way.
It's not hard to see how much easier to read and understand the piped version is, compared to the resulting code.
Also, you might be interested to know that piping can also be extended in really cool ways (similarly to how pattern-matching can be extended).
The way is to use piping macros (you may be noticing a theme here).
If you want to know more about those, feel free to check out Appendix D, where I review them in detail.
Oh, and before I forget, there is also a macro for doing reverse piping (which can be very useful in some situations).
Our previous example would look like this:
(<| (mix append_text "")
(interposed " ")
(each to_text)
elems)
Higher-Order Functions
I can't finish this chapter without talking about one of the coolest features in the world of functional programming.
So far, we have seen several control-flow mechanisms that could potentially exist in any language/paradigm, but now we'll talk about something native to the FP landscape.
You already know that in the world of functional programming, functions are first-class values.
That just means functions can be treated like other values (such as Ints, Bits and Texts).
You can create new functions at run-time, you can pass functions around as arguments to other functions and you can combine functions in arbitrary ways.
Well, we haven't really seen that in action yet.
It's time to put that theory into practice... with an example:
(def (iterate_list f list)
(All (_ a b) (-> (-> a b) (List a) (List b)))
(when list
{.#End}
{.#End}
{.#Item head tail}
{.#Item (f head) (iterate_list f tail)}))
This is a function that allows you to transform lists in arbitrary ways.
However, you may notice that we're seeing many new things.
For instance, what is that All thingie over there, and what does it do? Is it even a type?
Well, it's not technically a type.
It's actually a macro for creating types (in the same way that -> is a macro that creates types).
The difference is that All allows you to create universally-quantified types.
That's just a fancy way of saying that your types are not fixed to working in a particular way, but are flexible enough to allow some variability.
Here, it's being used to make a function that can takes lists with elements of any type (denoted by the type variable a), and can produce lists with elements of any other type (denoted by the type variable b), so long as you give it a function f, that transforms values of type a into values of type b.
That... is... mind-blowing!
In other programming languages, whenever you want to process the elements of a sequence (say, an array) you have to write something like a for loop with some index variable, some condition... and then the code for actually working with the data.
But here, we're pretty much defining a function that takes care of all the ceremony, so you just need to give it the operation you wish to perform on each element, and the data.
You could use it like this:
(iterate_list (n.* 5) (list 0 1 2 3 4 5 6 7 8 9))
... => (list 0 5 10 15 20 25 30 35 40 45)
Pretty cool, huh!
But this is just scratching the surface of what's possible.
As it turns out, higher-order functions (that is, functions which take or produce other functions) are at the foundation of many advanced techniques in functional programming.
Mastering this little trick will prove invaluable to you as you delve deeper into the mysteries of functional programming.
Alright.
We've seen quite a lot so far.
But, don't get complacent!
You've only seen the basics of Lux and the next chapters are going to expose some of the more advanced features the language.
Brace yourself, great power is coming!
See you in the next chapter!
Chapter 6: Types in detail
Where you will learn the truth behind types.
We've talked about Lux types already, but only in a very high-level way.
On this chapter, you'll see how types are constructed, and hopefully that will give you some insight to understand better the subjects of later chapters.
(type .public Type
(Rec Type
(Variant
{#Primitive Text (List Type)}
{#Sum Type Type}
{#Product Type Type}
{#Function Type Type}
{#Parameter Nat}
{#Var Nat}
{#Ex Nat}
{#UnivQ (List Type) Type}
{#ExQ (List Type) Type}
{#Apply Type Type}
{#Named Name Type})))
This is the type of types.
Crazy, right?
But as I've said before, Lux types are values like any other.
Type is a variant type, which just means that there are multiple options for type values.
Also, you may have noticed that Rec macro in the definition.
You need to add it whenever you're defining a recursive type that takes no parameters.
So, the definition of List doesn't need it, but the definition of Type does.
Let's go over each option for Type.
{#Primitive Text (List Type)}
This is what connects Lux's type-system with the host platform's.
These types represent classes (in the JVM) with their respective parameters, if they have them (as would be the case for ArrayList<Long> in the JVM).
{#Sum Type Type}
{#Product Type Type}
You may have noticed that none of those options are called #Variant or #Tuple.
The reason is that variants and tuples are just names for mathematical constructs called "sums" and "products".
Funny names, right?
Well, mathematicians see variants as a way of "adding" types and tuples as a way of "multiplying" types.
Of course, it's a bit difficult to picture that if you're thinking of numbers.
But a way to see variants is as an "OR" operation for types: you get this option OR that option.
Conversely, tuples are like an "AND" operation for types: you get this type AND that type.
But, you may be wondering: "why do #Sum and #Product only take 2 types, instead of a list like #Primitive does?"
Well, as it turns out, you don't need a list of types to implement variants and tuples, because you can actually chain #Sum and #Product with other instances of themselves to get the same effect.
What do I mean?
Well, let me show you. To the left, you'll see the type as it's written in normal Lux code, and to the right you'll see the type value it generates.
(Or) => Nothing
(Or Bit) => Bit
(Or Bit Int) => {#Sum Bit Int}
(Or Bit Int Real) => {#Sum Bit {#Sum Int Real}}
(Or Bit Int Real Char) => {#Sum Bit {#Sum Int {#Sum Real Char}}}
(And) => Any
(And Bit) => Bit
(And Bit Int) => {#Product Bit Int}
(And Bit Int Real) => {#Product Bit {#Product Int Real}}
(And Bit Int Real Char) => {#Product Bit {#Product Int {#Product Real Char}}}
You can see where this is going.
If I have a way to to pair up 2 types, and I can nest that, then I can chain things as much as I want to get the desired length.
What is a variant/tuple of 1 type? It's just the type itself; no pairing required.
This embedding means that [true 123 456.789 "X"] is the same as [true [123 456.789 "X"]], and the same as [true [123 [456.789 "X"]]].
It also means 5 is the same as [5], and [[5]], and [[[[[5]]]]].
As far as the compiler is concerned, there are no differences.
That might sound crazy, but there are some really cool benefits to all of this.
If you're curious about that, you can check out Appendix E for more information on how Lux handles this sort of stuff.
And what happens when the variant/tuple has 0 types?
That's when Nothing and Any come into play.
Nothing is a type that has no instances; which is to say, there's no expression which can yield a value of such a type.
It might seem oddd to have a type which has no instancces, but it can be useful to model computations which fail at runtime (thereby yielding no value).
So, another way of thinking of Nothing is as the type of failed/erroneous computations.
Any, on the other hand, is the opposite.
You can think of it as the super-type of all other types: the type of all values.
This means that not only (is Nat 123), but also (is Any 123).
This works because Any does not give you any specific information about a value, it only tells you that a value exists, regardless of what its specific type happens to be.
So, whenever a function accepts or returns a dummy value of some kind, Any is a good candidate for that.
An easy way to create values of type Any is with the empty tuple syntax [].
In the same way that you cannot have empty tuple types, you also cannot make empty tuples.
But Lux sees that syntax and just sticks some simple constant value in there for you.
You might think that dummy values are, well, dumb, but they show up all the time.
Consider the Maybe type:
(type .public (Maybe a)
(Variant
{#None}
{#Some a}))
The #Some tag holds values of type a, but what does #None hold? Nothing?
Well, Maybe is a variant, which means it's a #Sum, which looks like this:
{#Sum Type Type}
So we know that #None must hold something. But what?
Well, Anything, really.
So the type definition for Maybe is equivalent to this:
(type .public (Maybe a)
{#None Any} ... Alternatively, {#None []}
{#Some a})
If you don't care what value you store somewhere, then you can store Any value in there.
In practice, you can create instances of Maybe by writing this {#None []}, or {#None 123}, or just {#None}.
{#Function Type Type}
Now that we have discussed variant and tuple types, it shouldn't come as a surprise that a similar trick can be done with function types.
You see, if you can implement functions of 1 argument, you can implement functions of N arguments, where N > 1.
All I need to do is to embed the rest of the function as the return value to the outer function.
It might sound like this whole business of embedding variants, tuples and functions inside one another must be super inefficient; but trust me: Lux has taken care of that.
The Lux compiler features many optimizations that compile things down in a way that gives you maximum efficiency. So, to a large extent, these embedded encodings are there for the semantics of the language, but not as something that you'll pay for at run-time.
One of the cool benefits of this approach to functions is Lux's capacity to have partially applied functions.
Yep, that's a direct consequence of this theoretical model.
{#Parameter Nat}
This type is there mostly for keeping track of type-parameters in universal and existential quantification.
We'll talk about those later. But, suffice it to say that #Parameter helps them do their magic.
{#Var Nat}
These are type variables.
They are used during type-inference by the compiler, but they're also part of what makes universal quantification (with the All macro) able to adjust itself to the types you use it with.
Type-variables start unbound (which means they're not associated with any type), but once they have been successfully matched with another type, they become bound to it, and every time you use them afterwards it's as if you're working with the original type.
Type-variables, however, cannot be re-bound once they have been set, to avoid inconsistencies during type-checking.
{#Ex Nat}
An existential type is an interesting concept (which is related, but not the same as existential quantification).
You can see it as a type that exists, but is unknown to you.
It's like receiving a type in a box you can't open.
What can you do with it, then?
You can compare it to other types, and the comparison will only succeed if it is matched against itself.
It may sound like a useless thing, but it can power some advanced techniques.
{#UnivQ (List Type) Type}
This is what the All macro generates: universal quantification.
That (List Type) you see there is meant to be the context of the universal quantification.
It's kind of like the environment of a function closure, only with types.
The other Type there is the body of the universal quantification.
To understand better what's going on, let's transform the type of our iterate_list function from Chapter 5 into its type value.
(All (_ a b) (-> (-> a b) (List a) (List b)))
... =>
... {.#UnivQ {.#End} {.#UnivQ {.#End} (-> (-> {.#Parameter 3} {.#Parameter 1}) (List {.#Parameter 3}) (List {.#Parameter 1})}}
**Note**: I didn't transform the type entirely to avoid unnecessary verbosity.
As you can see, I do the same embedding trick to have universal quantification with multiple parameters.
Also, a and b are just nice syntactic labels that get transformed into #Parameter types.
The reason the type-parameters have those numeric IDs is due to a technique called [De Bruijn Indices](https://en.wikipedia.org/wiki/De_Bruijn_index).
{#ExQ (List Type) Type}
Existential quantification works pretty much the same way as universal quantification.
Its associated macro is Ex.
Where universal quantification works with type-variables, existential quantification works with existential types.
{#Apply Type Type}
This is the opposite of quantification.
#Apply is what you use to parameterize your quantified types; to customize them as you need.
With #Apply, (List Int) transforms into (#Apply Int List).
For multi-parameter types, like Dictionary (from library/lux/data/collection/dictionary), (Dictionary Text User) would become (#Apply User (#Apply Text Dictionary)).
As you can see, the nesting is slightly different than how it is for tuples, variant and functions.
{#Named Symbol Type}
#Named is less of a necessity and more of a convenience.
The type-system would work just fine without it, but users of the language probably wouldn't appreciate it while reading documentation or error messages.
#Named is what gives the name "List" to the List type, so you can actually read about it everywhere without getting bogged down in implementation details.
You see, Lux's type system is structural in nature, rather than nominal (the usual style in programming languages).
That means all that matters is how a type is built; not what you call it.
That implies 2 types with different names, but the exact same internal structure, would actually type-check in your code.
That may sound odd (if you come from Java or other languages with nominal types), but it's actually very convenient and enables you to do some pretty nifty tricks.
For more information on that structural types, head over to [Appendix E](appendix_e.md).
#Named gives Lux's type-system a bit of a nominal feel for the convenience of programmers.
Regarding error messages
When you get error messages from the type-checker during your coding sessions, types will show up in intuitive ways most of the time, with a few exceptions you might want to know about.
Existential types show up in error messages like +246 (where 246 is the ID of the type).
Whereas type-variables show up like -278.
Those types tend to show up when there are errors in the definition of some polymorphic function.
You may be tired of reading about types, considering that they are (to a large degree) an implementation detail of the language.
However, one of the key features of Lux is that types can be accessed and manipulated by programmers (often in macros) to implement various powerful features.
In the next chapter, you'll get acquainted with one such feature.
See you in the next chapter!
Chapter 7: Polymorphism a.k.a. interfaces and implementations
Where types and values collide.
You endured all that tedious talk about types; but it wasn't for nothing.
Now, you'll see types in action, as they take a new shape... and a new purpose.
Many programming languages have some kind of polymorphism system or module system.
You know what I'm talking about.
Object-oriented languages have classes with methods that can be overriden by their subclasses. The moment you call one of those methods on an object, the run-time system selects for you the correct implementation, based on the class hierarchy.
Or maybe you come from Haskell, where they have type-classes, that basically perform the same process, but during compilation. Types are checked, instances get picked, and the proper functions and constants get plugged-in.
Or maybe, you come from the world of ML (specially Standard ML), where they have a module system based on signatures and structures. In those systems, the function implementations you want don't get selected for you automatically (you have to pick them yourself), but you tend to have more control when it comes to choosing what to use.
The origin of Lux's polymorphism system is I... um... borrowed it from the SML guys.
I re-named signatures as interfaces and structures as implementations to give them more recognizable names.
But I also added my own little twist.
You see, polymorphism/module systems in programming languages tend to live in a mysterious world that is removed from the rest of the language.
It's a similar situation as with types.
Remember Lux's type system?
Most languages keep their types separate from their values.
Types are just some cute annotations you put in your code to keep the compiler happy.
Lux's types, on the other hand, are alive; for they are values.
Nothing stops you from using them, transforming them and analyzing them in ways that go beyond the language designer's imagination (that would be me).
Well, there's a similar story to tell about polymorphism/module systems.
The run-time/compiler chooses everything for you; and even when you choose for yourself, you're still somewhat limited in what you can do.
Implementations are not values, and there is a fundamental division between them and the rest of the language.
But not in Lux.
Lux's polymorphism system is actually based on regular types and values.
And because types are values, that means it's just turtles values all the way down.
But, how does it work?
Read on!
Interfaces
They provide a description of the functionality expected of proper implementations.
They have a list of expected member values/functions, with their associated types.
Here's an example:
(type .public (Order a)
(Interface
(is (Equivalence a)
equivalence)
(is (-> a a Bit)
<)))
That interface definition comes from the library/lux/abstract/order module, and it deals with ordered types; that is, types for which you can compare their values in ways that imply some sort of sequential order.
It's polymorphic/parameterized because this interface must be able to adapt to any type that fits its requirements.
Also, you may notice that it has a member called equivalence, of type (Equivalence a).
The reason is that interfaces can expand upon (or be based on) other interfaces (such as Equivalence).
How do interfaces differ from types?
They don't.
They're actually implemented as types.
Specifically, as tuple/record types.
You see, if I can create a record type with one field for every expected definition in a interface, then that's all I need.
Implementations
They are the other side of the coin.
If interfaces are record types, then that means implementations must be actual records.
Let's take a look at how you make one:
(def .public order
(Order Frac)
(implementation
(def equivalence ..equivalence)
(def < ..<)))
This implementation comes from library/lux/math/number/frac.
As you may notice, implementations have names; unlike in object-oriented languages where the "implementation" would just be the implemented methods of a class, or Haskell where instances are anonymous.
For implementations, the convention is just to name them as lower-cased versions of the interfaces they implement.
Here is another example, from the library/lux/data/collection/list module:
(def .public monoid
(All (_ a)
(Monoid (List a)))
(implementation
(def identity
{.#End})
(def (compose xs ys)
(when xs
{.#End} ys
{.#Item x xs'} {.#Item x (compose xs' ys)}))))
The reason why implementations have names (besides the fact that they are definitions like any other), is that you can usually construct multiple valid implementations for the same combination of interfaces and parameter types.
That would require you to distinguish each implementation in some way in order to use it.
This is one cool advantage over Haskell's type-classes and instances, where you can only have one instance for any combination of type-class and parameter.
Haskellers often resort to _"hacks"_ such as using newtype to try to get around this limitation.
The upside of having the run-time/compiler pick the implementation for you is that you can avoid some boilerplate when writing polymorphic code.
The upside of picking the implementation yourself is that you get more control and predictability over what's happening (which is specially useful when you consider that implementations are first-class values).
What's the big importance of implementations being first-class values?
Simple: it means you can create your own implementations at run-time based on arbitrary data and logic, and you can combine and transform implementations however you want.
Standard ML offers something like that by a mechanism they call "functors" (unrelated to a concept of "functor" we'll see in a later chapter), but they are more like magical functions that the compiler uses to combine structures in limited ways.
In Lux, we dispense with the formalities and just use regular old functions and values to get the job done.
How to use implementations
We've put functions and values inside our implementations.
It's time to get them out and use them.
There are 2 main ways to use the stuff inside your implementations: use and at.
Let's check them out.
... Opens an implementation and generates a definition for each of its members (including nested members).
... For example:
(use "i::[0]" library/lux/math/number/int.order)
... Will generate:
(def .private i::= (at library/lux/math/number/int.order =))
(def .private i::< (at library/lux/math/number/int.order <))
The use macro serves as a directive that creates private/un-exported definitions in your module for every member of a particular implementation.
You may also give it an optional aliasing pattern for the definitions, in case you want to avoid any name clash.
You might want to check out [Appendix C](appendix_c.md) to discover a pattern-matching macro version of `use` called `^open`.
... Allows accessing the value of a implementation's member.
(is (-> Int Text)
(at library/lux/math/number/int.decimal encoded))
... Also allows using that value as a function.
(at library/lux/math/number/int.decimal encoded +123)
... => "+123"
at is for when you want to use individual parts of a implementation immediately in your code, instead of opening them first.
Psss! Did you notice `at` is _piping compatible_?
Also, you don't really need to worry about boilerplate related to using implementations.
There is a module called library/lux/type/implicit which gives you a macro called a/an for using implementations without actually specifying which one you need.
Psss! This macro also has 2 shorter aliases: `a` and `an`.
The macro infers everything for you based on the types of the arguments, the expected type of the expression, and the implementations available in the environment.
For more information about that, head over to Appendix F to read more about that.
Implementations as values
I can't emphasize enough that implementations are values.
And to exemplify it for you, here's a function from the library/lux/abstract/monad module that takes in an implementation (among other things) and uses it within its code:
(def .public (each monad f xs)
(All (_ M a b)
(-> (Monad M) (-> a (M b)) (List a) (M (List b))))
(when xs
{.#End}
(at monad in {.#End})
{.#Item x xs'}
(do monad
[y (f x)
ys (each monad f xs')]
(in {.#Item y ys}))))
Monad is an interface and the each function takes arbitrary Monad implementations and can work with any of them without any issue.
Interfaces and implementation are the main mechanism for writing ad-hoc polymorphic code in Lux, and they allow flexible and precise control over polymorphism.
It may be the case that in the future Lux includes new mechanisms for achieving the same goals (I believe in having variety), but the spirit of implementing things in terms of accessible values anybody can manipulate will likely underlie every such mechanism.
Now that we've discussed interfaces and implementations, it's time to talk about a very special family of interfaces.
See you in the next chapter!
Chapter 8: Functors and monads
Where I will try to explain something really confusing, and you'll pretend you understand to avoid hurting my feelings.
OK. It's time to get serious.
The following topics are known to be troublesome to teach, so I can only promise you that I will try really, really hard not too say something confusing (or stupid).
Functors
Functors and monads are both mathematical concepts that are prevalent in Category Theory.
You may have heard of it before.
It's a branch of abstract mathematics that many are drawing inspiration from when developing new tools and techniques for functional programming.
But I will not go down the route of explaining things to you from a mathematical perspective... as I'm not confident that's going to work.
Imagine that you have some data (maybe an Int, or a Text).
You can work with that data: you can pass it around, apply functions to it, print it to the console/terminal, or pattern-match against it.
Well, imagine a Functor as some kind of wrapper on your data.
You can still access what's inside, but the wrapper itself offers special superpowers, and each wrapper is different.
For instance, there's one wrapper that allows us to have (or not to have) our data.
Schrodinger's wrapper (although most people prefer to call it Maybe).
That wrapper happens to be a type, as all Functor wrappers are types.
But not just any type. You see, functors have requirements.
(type .public (Functor f)
(Interface
(is (All (_ a b)
(-> (-> a b) (f a) (f b)))
each)))
This is the Functor interface, from library/lux/abstract/functor.
As you can see, it only has a single member: the each function.
The parameter type f is very special, because instead of being a simple type (like Int or Text), it's actually a parameterized type (with a single parameter).
That explains why it's being used the way it is in the type of each.
Not every parameterized type can be a Functor, but if the type is something that you can open to work with its inner elements, then it becomes a good candidate.
And you would be surprised how many things fit that requirement.
Remember that Maybe type we talked about?
Let's see how it plays with Functor.
(type .public (Maybe a)
(Variant
{.#None}
{.#Some a}))
We've seen Maybe before, but now we can analyse out how it's implemented.
By the way, it lives in the `library/lux` module, so you don't need to import anything.
Here is its Functor implementation.
(def .public functor
(Functor Maybe)
(implementation
(def (each f ma)
(when ma
{.#None} {.#None}
{.#Some a} {.#Some (f a)}))))
... This one lives in the library/lux/data/maybe module, though.
We'll know how everything fits if we fill in the blanks for each's type:
(All (_ a b)
(-> (-> a b) (Maybe a) (Maybe b))
So, the job of each here is to take a Maybe containing some a value, and somehow transform it into a b, without escaping the Maybe.
By looking at the Functor implementation, we can see how this works out.
We can actually pattern-match against the entire input and handle the different cases, using the given function to transform our a into a b.
Not that hard.
Oh, and remember our iterate_list function from chapter 5?
Turns out, that's just the Functor implementation from library/lux/data/collection/list:
(def .public functor
(Functor List)
(implementation
(def (each f ma)
(when ma
{.#End} {.#End}
{.#Item a ma'} {.#Item (f a) (each f ma')}))))
Not bad.
In the case of List, the wrapper superpower it provides is the capacity to handle multiple values as a group.
This can be used for some really cool techniques; like implementing non-deterministic computations by treating every list item as a branching value (but let's not go down that rabbit-hole for now).
The power of Functors is that they allow you to decorate your types with extra functionality.
You can still access the inner data and the each function will take advantage of the wrapper's properties to give you that extra power you want.
You can implement things like stateful computations, error-handling, logging, I/O, asynchronous concurrency and many other crazy things with the help of Functors.
However, to make them really easy to use, you might want to add some extra functionality.
Monads
One thing you may have noticed about the Functor interface is that you have a way to operate on functorial values, but you don't have any standardized means of creating them.
I mean, you can use the list and library/lux/data/collection/list.partial macros to create lists and the .#None and .#Some tags for Maybe, but there is no unified way for creating any functorial value.
Well, let me introduce you to Monad:
(type .public (Monad m)
(Interface
(is (Functor m)
functor)
(is (All (_ a)
(-> a (m a)))
in)
(is (All (_ a)
(-> (m (m a)) (m a)))
conjoint)))
This interface extends Functor with both the capacity to wrap a normal value in a functorial structure, and to join 2 layers of functorial structure into a single, conjoint one.
Sweet!
Wrapping makes working with functors so much easier because you don't need to memorize a bunch of tags, macros or functions in order to create the structures that you need.
And being able to join layers of functorial structure allows you to write dynamic computations which make use of the functorial structure and that can depend on the value of previous functorial computations.
To get a taste for it, let's check out another functorial type.
Remember what I said about error-handling?
(type .public (Try a)
(Variant
{#Failure Text}
{#Success a}))
This type expresses errors as Text values (and it lives in the library/lux/control/try module).
Here are the relevant Functor and Monad implementations:
(def .public functor
(Functor Try)
(implementation
(def (each f ma)
(when ma
{#Failure msg}
{#Failure msg}
{#Success datum}
{#Success (f datum)}))))
(def .public monad
(Monad Try)
(implementation
(def functor ..functor)
(def (in a)
{#Success a})
(def (join mma)
(when mma
{#Failure msg}
{#Failure msg}
{#Success ma}
ma))))
If you listen to functional programmers, you'll likely get the impression that the invention of monads rivals the invention of the wheel.
It is this incredibly powerful and fundamental abstraction for a lot of functional programs.
The thing about Monad is that, with it, you can use each functions that also generate wrapped values (and take advantage of their special properties), and then you can collapse/merge/combine those values into a "conjoint" value by using the conjoint function.
Let's see that in action:
(.require
[library
[lux (.except)
[data
[collection
["[0]" list]]]]])
(use list.monad)
(def foo
(|> (list 1 2 3 4)
(each (list.repeated 3))
conjoint))
... The value of 'foo' is:
(list 1 1 1 2 2 2 3 3 3 4 4 4)
It's magic!
Not really. It's just the Monad for List:
(def .public functor
(Functor List)
(implementation
(def (each f ma)
(when ma
{.#End}
{.#End}
{.#Item a ma'}
{.#Item (f a) (each f ma')}))))
(def .public mix
(Mix List)
(implementation
(def (mix f init xs)
(when xs
{.#End}
init
{.#Item x xs'}
(mix f (f x init) xs')))))
(def .public monoid
(All (_ a) (Monoid (List a)))
(implementation
(def identity
{.#End})
(def (composite xs ys)
(when xs
{.#End}
ys
{.#Item x xs'}
{.#Item x (compose xs' ys)}))))
(use "[0]" ..monoid)
(def .public monad
(Monad List)
(implementation
(def functor ..functor)
(def (in a)
{.#Item a {.#End}})
(def (conjoint list_of_lists)
(mix composite
identity
(reversed list_of_lists)))))
... The mix function is for doing incremental iterative computations.
... Here, we're using it to build the total output list by composing/concatenating all the input lists in our `list_of_lists`.
Monads are incredibly powerful, since being able to use the special power of our Functor while applying functions to each list item allows us to layer that power in complex ways.
But... you're probably thinking that writing a bunch of eachs followed by conjoints is a very tedious process. And, you're right!
If functional programmers had to subject themselves to that kind of tedium all the time, they'd probably not be so excited about monads.
Time for the VIP treatment.
The do macro
These macros always show up at the right time to saves us from our hurdles!
(.require
[library
[lux (.except)
[data
["[0]" maybe]]]])
... Macro for easy concatenation of monadic operations.
(do maybe.monad
[x (f0 123)
.let [y (f1 x)] ... .let enables you to use full-featured let-expressions within do
z (f2 y)]
(in (f3 z)))
The do macro allows us to write monadic code with great ease (it's almost as if we're just making let bindings).
Just tell it which Monad implementation you want, and it will write all the steps in your computation piece by piece using each and conjoint without you having to waste your time with all the boilerplate.
Finally, whatever you write as the body of the do, it must result in a functorial/monadic value (in this case, a Maybe value).
Remember: A conjoint value may collapse/merge/combine layers of the Functor, but it never escapes it.
Functors and Monads have a bad reputation for being difficult to understand, but hopefully I didn't botch this explanation too much.
Personally, I think the best way to understand Functors and Monads is to read different implementations of them for various types (and maybe write a few of your own).
For that, feel free to peruse [the Lux Standard Library](https://github.com/LuxLang/lux/tree/master/documentation/library/standard) at your leisure.
This is the sort of thing that you need to learn by intuition and kind of get the feel for.
Hopefully, you'll be able to get a feel for them in the next chapters, because we're going to be exploring a lot of monads from here on.
So, buckle-up, cowboy. This ride is about to get bumpy.
See you in the next chapter!
Chapter 9: Meta-programming
Where we take a peek behind the curtains of the compiler.
Metaprogramming is the art of making programs... that make programs.
There are many techniques and tools for achieving this, but one that is very familiar to Lisp fans is to use macros to generate code at compile-time.
However, we're not going to talk about macros in this chapter.
Instead, I'll reveal the infrastructure that makes macros possible, and we'll discuss macros in the next chapter.
The Lux type
Yeah, I'm aware that it's weird there's a type with the same name as the language, but I couldn't figure out a better name.
The Lux compiler was designed to integrate very well with the language itself.
Most compilers are just programs that take source code and emit some binary executable or some byte-code.
But the Lux compiler opens itself for usage within Lux programs and provides Lux programmers with a wealth of information.
The Lux type enters the stage.
(type .public Lux
(Rec Lux
(Record
[#info Info
#source Source
#location Location
#current_module (Maybe Text)
#modules (List [Text Module])
#scopes (List Scope)
#type_context Type_Context
#expected (Maybe Type)
#seed Nat
#scope_type_vars (List Nat)
#extensions Any
#eval (-> Type Code (-> Lux (Either Text [Lux Any])))
#host Any])))
By the way, the `Lux` type and other weird types you may not recognize there are all defined in the `library/lux` module.
Check [the documentation for the Standard Library](https://github.com/LuxLang/lux/tree/master/documentation/library/standard) for more details.
The Lux type represents the state of the Lux compiler at any given point.
It is not a reflection of that state, or a subset of it.
It is the state of the Lux compiler; and, as you can see, it contains quite a lot of information about compiled modules, the state of the type-checker, the lexical and global environments, and more.
Heck, you can even access the yet-unprocessed source code of a module at any given time.
That's pretty neat.
You can actually write computations that can read and even modify (careful with that one) the state of the compiler.
This turns out to be massively useful when implementing a variety of powerful macros.
For example, remember the use and at macros from chapter 7?
They actually look up the typing information for the structures you give them to figure out the names of members and generate the code necessary to get that functionality going.
And that is just the tip of the iceberg.
The possibilities are really vast when it comes to using the information provided by the Lux compiler state.
The Meta type
But, how do I use it?
Well, that is where the Meta type and the library/lux/meta module come into play.
The library/lux/meta module houses many functions for querying the Lux compiler state for information, and even to change it a little bit (in safe ways).
I won't go into detail about what's available, but you'll quickly get an idea of what you can do if you read the documentation for it in the Standard Library.
However, one thing I will say is that those functions rely heavily on the Meta type, which is defined thusly:
(type .public (Meta a)
(-> Lux (Either Text [Lux a])))
The `Meta` type is defined in the `library/lux` module, although most functions that deal with it are in the `library/lux/meta` module.
The Meta type has a Functor, and a Monad, but they are a bit rather complicated.
You saw some Functor/Monad examples in the last chapter, but this is more colorful.
Meta instances are functions that given an instance of the Lux compiler state, will perform some calculations which may fail (with an error message); but if they succeed, they yield a value, plus a (possibly updated) instance of the Lux compiler state.
Lux metaprogramming is based heavily on the Meta type, and macros themselves rely on it for many of their functionalities, as you'll see in the next chapter.
Where do Lux instances come from?
Clearly, Lux instances are data, but the compiler is not available at all times.
The compiler is only ever present during... well... compilation.
And that is precisely when all of your Lux-dependant code will execute.
Basically, in order for you to get your hands on that sweet compiler information, your code must be run at compile-time.
But only macro code can ever do that, so you will have to wait until the next chapter to learn the conclusion to this story.
This chapter feels a little empty because the topic only makes sense within the context of macros.
But macros by themselves are a huge subject, and involve more machinery than you've seen so far.
However, I wanted to give you a taste of what's possible in order to whet your appetite, while keeping the chapter focused.
In the next chapter, I'll complete this puzzle, and you'll be given access to a power greater than you've ever known (unless you've already been a lisper for a while).
See you in the next chapter!
Chapter 10: Code and macros
Where magic turns into science.
I've talked about many macros in this book.
There's a macro for this and a macro for that.
You use macros for defining stuff, for making types and functions and lists, for doing pattern-matching, and for control-flow.
There's a macro for everything.
Yet, I haven't even shown a macro being defined yet.
Quiet your mind, young grasshopper. You're about to be enlightened.
But first, you need to learn a few things.
The AST
The word AST stands for Abstract Syntax Tree.
An AST is a representation of the syntax of a programming language, and compilers use them for the sake of analyzing the source-code (like, by type-checking it), and then generating the binary/byte-code output.
You might think that's none of your business.
Only compiler writers have to worry about that stuff, right?
Oh, you have much to learn, young grasshopper.
You see, the power of macros lies in the fact that (to some extent) users of the language can play the role of language designers and implementers.
Macros allow you to implement your own features in the language and to have them look and feel just like native features.
I mean, beyond the native syntax for writing numbers, text, variants, tuples and records, every single thing you have written so far has been macros.
.require import statements? Yep, macros.
Definition statements? Yep, macros.
Function expressions? Yep, macros.
And you'd have never suspected those weren't native Lux features had I not told you they were macros.
Now, just imagine making your own!
But macros work with the Lux AST, so that's the first thing you need to master.
Check it out:
(type .public Location
(Record
[#module Text
#line Nat
#column Nat]))
(type .public (Ann m v)
(Record
[#meta m
#datum v]))
(type .public (Code' w)
(Variant
{#Bit Bit}
{#Nat Nat}
{#Int Int}
{#Rev Rev}
{#Frac Frac}
{#Text Text}
{#Symbol Symbol}
{#Form (List (w (Code' w)))}
{#Variant (List (w (Code' w)))}
{#Tuple (List (w (Code' w)))}))
(type .public Code
(Ann Location (Code' (Ann Location))))
The Code type is the one you'll be interacting with, but all it does is wrap (recursively) the incomplete Code' type, giving it some meta-data Annotations to know where each AST node comes from in your source-code.
The real magic is in the Code' type, where you can see all the alternative syntactic elements.
The Symbol type (from the library/lux module), is just a [Text Text] type.
The first part holds the module/prefix of the identifier/symbol, and the second part holds the name itself.
So library/lux/data/collection/list.reversed becomes ["library/lux/data/collection/list" "reversed"], and each becomes ["" "each"].
`list.reversed` would become `["library/lux/data/collection/list" "reversed"]` anyway, because aliases get resolved prior to analysis and macro expansion.
Forms are (syntactic structures delimited by parentheses), variants are {syntactic structures delimited by braces}, and tuples are [syntactic structures delimited by brackets].
Quotations
We know everything we need to extract information from the Code type, but how do we build Code values?
Do we have to build it with our bare hands using variants and tuples?
That sounds... exhausting.
Well, we don't have to. There are actually many nice tools for making our lives easier.
One nice resource within our reach is the library/lux/macro/code module, which contains a variety of functions for building Code values, so we don't have to worry about locations and tags and all that stuff.
But, even with that, things would get tedious.
Imagine having to generate an entire function definition (or something even larger), by having to call a bunch of functions for every small thing you want.
Well, don't fret. The Lux Standard Library already comes with a powerful mechanism for easily generating any code you want and you don't even need to import it (i.e. it's in the library/lux module).
... Quotation as a macro.
(' "YOLO")
Quotation is a mechanism that allows you to write the code you want to generate, and then builds the corresponding Code value.
The ' macro is the simplest version, which does exactly what I just described.
This would turn the text "YOLO" into [[.#module "" .#line 0 .#column 0] {.#Text "YOLO"}].
If you want to know what that would look like with the tools at library/lux/macro/code, it would be: (text "YOLO").
The beautiful thing is that (' (you can use the "'" #macro [to generate {arbitrary code} without] worrying (about the "complexity"))).
... Hygienic quasi-quotation as a macro.
... Unquote (,) and unquote-splice (,*) must also be used as forms.
... All unprefixed symbols will receive their parent module's prefix if imported; otherwise will receive the prefix of the module on which the quasi-quote is being used.
(` (def (, name)
(function ((, name) (,* args))
(, body))))
This is a variation on the ' macro that allows you to do templating with the code you want to generate.
Everything you write will be generated as is, except those forms which begin with , or ,*.
, means: evaluate this expression and use its Code value.
,* means: the value of this expression is a list of Codes, and I want to splice all of them in the surrounding Code node.
With these tools, you can introduce a lot of complexity and customization into your code generation, which would be a major hassle if you had to build the Code nodes yourself.
You may be wondering what does "hygienic" means in this context.
It just means that if you use any identifier in your template which may refer to an in-scope definition or local variable, the identifier will be resolved to it.
Any identifier that does not correspond to any known in-scope definition or variable will trigger a compile-time error.
This ensures that if you make a mistake writing your template code, it will be easy to spot during development.
Also, it will be harder to collide (by mistake) with user code if you, for instance, write the code for making a local variable named `foo`, and then the person using your macro uses a different `foo` somewhere in their code.
... Unhygienic quasi-quotation as a macro.
... Unquote (,) and unquote-splice (,*) must also be used as forms.
(`' (def (, name)
(function ((, name) (,* args))
(, body))))
Finally, there is this variation, which removes the hygiene check.
Out of the 3 variations, the one you'll most likely use is the 2nd one, since it provides both safety and power.
Macros
Now that you know how to generate code like a pro, it's time to see how macros get made.
First, let's check the type of macros:
(type .public Macro
(primitive "#Macro"))
That does not look particularly useful.
What the hell is a "#Macro"?
Fundamentally, all macros are functions.
However, the compiler cannot treat them as normal functions because they must be applied to code at compile-time, rather than run-time.
For this reason, the Lux compiler must have some way to identify macros as distinct from functions.
It does so by labelling macros (type-wise) with this funky type.
There is, however, another type which elucidates what is going on with macros.
(type .public Macro'
(-> (List Code) (Meta (List Code))))
You might remember from the previous chapter that you can only access the Lux compiler state inside of macros.
Now, you can see how everything connects.
You define macros by using the macro macro (so meta...):
(def .public symbol
(macro (_ tokens)
(when tokens
(list [_ (.#Symbol [module name])])
(at meta.monad in (list (` [(, (code.text module)) (, (code.text name))])))
_
(meta.failure "Wrong syntax for 'symbol'."))))
Here's another example:
(def .public else
(macro (_ tokens state)
(when tokens
(list else maybe)
(let [g!temp (macro.symbol "")]
(.#Right [state (.list (` (when (, maybe)
{.#Some (, g!temp)}
(, g!temp)
{.#None}
(, else))))]))
_
(.#Left "Wrong syntax for else"))))
You may want to read [Appendix C](appendix_c.md) to learn about the pattern-matching macros used in these examples.
As you can see, I'm using both quotation and the functions from the library/lux/macro/code module to generate code here.
I'm also using the symbol function from library/lux/macro, which generates unique symbols for usage within code templates in order to avoid collision with any code provided by the user of the macro.
A macro receives the raw List of Code tokens and must process them manually to extract any information it needs for code generation.
After that, a new List of Code tokens must be generated.
If any macros are used in the output, they will be expanded further until only primitive/native syntax remains that the Lux compiler can then analyze and compile.
You may be wondering what is the relationship between the `Macro` and `Macro'` types.
When you define a macro, you define it as a function, which is to say, a `Macro'` type.
But once it has been defined, it gets re-labelled as a `Macro`, so that way the Lux compiler can distinguish it from other functions.
This is all done for you by the `macro` macro, so there's no need to worry about it.
You have learned how to use one of the greatest superpowers that Lux has to offer.
But, if you're like me, you might be getting the nagging feeling that something is not right here.
I mean, if I have to pattern-match against the code I receive; what happens when my macros have complex inputs?
Clearly, analyzing the input code is far more difficult than generating it with the quoting macros.
Don't worry about it.
Because in the next chapter, you will learn a more sophisticated method of macro definition that will make writing complex macros a breeze.
See you in the next chapter!
Chapter 11: Syntax macros
Where science turns into magic once more.
You've now learned how to create your own macros to make your own custom syntax, and the features involved.
I would advice you to take a look at the many macros in the Lux Standard Library for inspiration as to what can be accomplished.
In the meantime, let's find out how to take our macro chops to the next level.
The library/lux/control/parser/code module houses some powerful tools.
For starters, it's the home of the (code) Parser type:
(type .public Parser
(//.Parser (List Code)))
Which is based on the generic Parser type from the library/lux/control/parser module:
(type .public (Parser s a)
(-> s (Try [s a])))
**Note**: This is also a functorial/monadic type.
Parser (from library/lux/control/parser/code) is the type of code-parsers: parsers which analyze Code nodes to extract arbitrary information.
The Parser type works with streams of inputs instead of single elements, and it often consumes some of those inputs, which is why the output involves an updated list of Codes.
There are many such code-parsers (and combinators) in the library/lux/control/parser/code module, and you should definitely take a look at what's available in the documentation.
Then, in the library/lux/macro/syntax module, there is a mechanism for defining macros: the syntax macro.
... A more advanced way to define macros than 'macro'.
... The inputs to the macro can be parsed in complex ways through the use of syntax parsers.
... The macro body is also (implicitly) run in the Meta monad, to save some typing.
... Also, the compiler state can be accessed through the *lux* binding.
(def .public object
(syntax (_ [.let [imports (class_imports *lux*)
class_vars (list)]
super (opt (super_class_decl^ imports class_vars))
interfaces (tuple (some (super_class_decl^ imports class_vars)))
constructor_args (constructor_args^ imports class_vars)
methods (some (overriden_method_def^ imports))])
(let [def_code ($_ text#composite "anon-class:"
(spaced (list (super_class_decl$ (maybe.else object_super_class super))
(with_brackets (spaced (list#each super_class_decl$ interfaces)))
(with_brackets (spaced (list#each constructor_arg$ constructor_args)))
(with_brackets (spaced (list#each (method_def$ id) methods))))))]
(in (list (` ((, (code.text def_code)))))))))
This example is a macro for making anonymous _JVM_ classes that lives in `lux/ffi`.
The difference between macro and syntax is that syntax allows you to parse, in a structured manner, the inputs to your macro, thereby reducing considerably the complexity necessary for making big macros.
Also, because you're using code-parsers for the hard work, you can write reusable parsers that you can share throughout your macros, if you want to have common syntax.
You can even compose your parsers, or use parsers from someone else's library.
There are already some small modules under `library/lux/macro/syntax/` which house some reusable code-parsers and code-generators.
Additionally, syntax binds the Lux value on a variable called *lux*, so you can use it during your parsing.
What do those code-parsers look like?
Here is an example:
... Taken from library/lux/math/infix.
(.require
[library
[lux (.except)
[abstract
[monad (.only do)]]
[control
["<>" parser (.use "[1]#[0]" functor)]]
[data
["[0]" product]
[collection
["[0]" list (.use "[1]#[0]" mix)]]]
[meta
["[0]" code
["<[1]>" \\parser (.only Parser)]]
[macro
[syntax (.only syntax)]
["[0]" code]]]
[math
[number
["n" nat]
["i" int]]]]])
(type Infix
(Rec Infix
(Variant
{#Const Code}
{#Call (List Code)}
{#Unary Code Infix}
{#Binary Infix Code Infix})))
(def literal
(Parser Code)
($_ <>.either
(<>#each code.bit <code>.bit)
(<>#each code.nat <code>.nat)
(<>#each code.int <code>.int)
(<>#each code.rev <code>.rev)
(<>#each code.frac <code>.frac)
(<>#each code.text <code>.text)
(<>#each code.symbol <code>.symbol)))
(def expression
(Parser Infix)
(<| <>.rec (function (_ expression))
($_ <>.or
..literal
(<code>.form (<>.many <code>.any))
(<code>.tuple (<>.and <code>.any expression))
(<code>.tuple (do <>.monad
[init_subject expression
init_op <code>.any
init_param expression
steps (<>.some (<>.and <code>.any expression))]
(in (list#mix (function (_ [op param] [_subject _op _param])
[{#Binary _subject _op _param} op param])
[init_subject init_op init_param]
steps))))
)))
And here are some examples of syntax macros:
... Also from library/lux/math/infix.
(def (prefix infix)
(-> Infix Code)
(when infix
{#Const value}
value
{#Call parts}
(code.form parts)
{#Unary op subject}
(` ((, op) (, (prefix subject))))
{#Binary left op right}
(` ((, op) (, (prefix right)) (, (prefix left))))))
(def .public infix
(syntax (_ [expr ..expression])
(in (list (..prefix expr)))))
(def .public ^stream&
(syntax (_ [patterns (<code>.form (<>.many <code>.any))
body <code>.any
branches (<>.some <code>.any)])
(with_symbols [g!stream]
(let [body+ (` (let [(,* (|> patterns
(list#each (function (_ pattern)
(list (` [(, pattern) (, g!stream)])
(` ((,! //.result) (, g!stream))))))
list#conjoint))]
(, body)))]
(in (list& g!stream body+ branches))))))
(def .public cond>
(syntax (_ [_ _reversed_
prev <code>.any
else body^
_ _reversed_
branches (<>.some (<>.and body^ body^))])
(with_symbols [g!temp]
(in (list (` (let [(, g!temp) (, prev)]
(cond (,* (do list.monad
[[test then] branches]
(list (` (|> (, g!temp) (,* test)))
(` (|> (, g!temp) (,* then))))))
(|> (, g!temp) (,* else))))))))))
This may be a short chapter, but not because its subject is small.
The opportunities that code-parsers open are fantastic, as it puts within your reach macros which would otherwise be much harder to implement correctly.
Don't worry about complex inputs: your macros can implement entire new embedded programming languages if you want them to.
Code-parsers can generate any data-type you want, so you can easily translate the information in the input syntax to whatever data-model you need.
But, now that we've spent 3 chapters about metaprogramming in Lux, I think it's fair that we clear our minds a little by looking at other subjects.
You're going to learn how to go beyond Lux and interact with everything and everyone.
See you in the next chapter!
Chapter 12: I/O
Where you will learn how to interact with the outside world.
I/O (short for input and output) is a very important subject in programming.
Arguably, it's the only reason why we make programs in the first place.
Software wouldn't be very useful if we couldn't use files, or interact with foreign devices (like sensors, or other computers), or interact with our users through GUIs.
I/O is fundamental to software development; but, like every fundamental thing in programming, there are many approaches to it, and some of them are in opposition to others.
I would say there are 2 main schools of thought when it comes to the role of I/O in computer programs.
Implicit I/O
These guys say that all programs are fundamentally about the I/O.
Everything, in one way or another, revolves around I/O, and it is a pervasive force that defines what computing is about from its very foundations.
Most programming languages you may be familiar with subscribe to this idea (either by choice of the language designer(s) or by simply following standard practice).
Here, we see operations which have side-effects (such as printing to standard output, or reading files, or connecting to another computer over the network) be treated like operations which lack them (like adding two numbers), with the side-effects being seen as some kind of magical property exhibited by those operations, which neither the language nor its libraries are obligated to handle or track in any special way.
Side-effects tend to be mentioned in documentation, but the lack of separation means that programmers must be wary of what they're using if the want to avoid the unfortunate consequences of careless coding.
A common pitfall happens to involve concurrency, where operations being executed on multiple threads, and having access to common resources, can cause data-corruption and other problems during run-time.
However, programmers who subscribe to this philosophy don't tend to see that as a sign that their languages or tools lack anything, but that programmers need to be careful and mindful of what they code, and that the effectful nature of programming is something to be embraced, rather than constrained.
Explicit I/O
These guys are a very different breed.
Just like static typing is used to impose some order in programs and to make explicit that which only lived in the minds (and comments) of programmers; explicit I/O acknowledges that effectful computations are fundamentally different from pure computations (that is, computations which just process their inputs and calculate their outputs).
Making I/O explicit, rather than being a denial of the very nature of programs, is an acknowledgement of this reality, and a reification of something which only used to live in the minds of programmers (and their comments).
By making I/O explicit, it can be separated from pure computations (thus, avoiding many common mistakes); it can be isolated and handled in safe places (thus avoiding its often pervasive nature); and it can be better integrated into the design of programs by making it a more concrete element.
In a way, it can even be said that immutable data-types (a staple of functional programming that Lux embraces) are just another form of explicit I/O.
Being able to change data after it was created can easily propagate changes beyond your local scope and affect threads and data-structures you aren't even aware of in ways that may be troublesome.
But, just like having immutable data-types makes change explicit and a concrete part of one's design, making I/O explicit achieves a similar goal, with regards to one's interactions with the outside world.
While immutable data-types deal with the mechanics of the inner world of programs, explicit I/O deals with interaction with the outside world; from the very near to the very far.
Lux chooses explicit I/O as its underlying model.
It may seem odd that I have to justify the choice of explicit _I/O_ in Lux; but, at the time of this writing, implicit _I/O_ is the industry standard, with many even doubting the benefits of an alternative approach.
The only _major_ language which also adopts this model is Haskell (from which Lux takes _much inspiration_), in its efforts to maintain theoretical purity.
How does explicit I/O work?
Now that I have (hopefully) convinced you that this is a good idea, how does it work?
Well, by using types; of course!
If you head to the library/lux/control/io module, you will find the following type definition:
(primitive .public (IO a)
...
)
IO is a type that embodies this model, by making it something that the type-system can recognize and work with.
If you explore the library/lux/control/io module, you'll find the means to run! (or execute) these IO operations.
The io macro, on the other hand, gives you the way to label effectful code as such, by wrapping it inside the IO type.
However, it's unlikely you'll need to use the `io` macro yourself very often, as some tools we'll see later can take care of that for you.
What IO operations are available?
Sadly, this is one area where Lux (currently) falls short.
There are 2 reasons for that:
- Lux's cross-platform ideals.
- Lux's youth.
Different platforms (e.g. JVM, Node.js, web browsers, CLR) offer different tools for doing I/O, which makes providing cross-platform I/O functionality a bit of a challenge.
As Lux grows (as a language and as a community), many of the gaps regarding I/O will be filled and there will (hopefully) be some sort of standard I/O infrastructure that will be relatively platform-independent.
In the meantime, I/O will depend on the capabilities of the host platform, accessed through the mechanisms Lux provides for doing host inter-operation (which is, coincidentally, the subject of the next chapter).
This is not as bad as it sounds; and one way or another, programmers will always want to reach to their host platforms to access features beyond what Lux can provide.
And this is fine.
Rather than trying to be what everybody needs and wants all the time, Lux chooses to focus on the subset of things it can do really well, and leaves the rest on the hands of the platform, and the clever programmers who use both.
But, I must confess, there is _one_ way of doing _I/O_ Lux **does provide** in a cross-platform way, which is the `library/lux/debug.log!` function (which prints to standard output).
However, `library/lux/debug.log!` has type `(-> Text Any)`, instead of the expected `(-> Text (IO Any))`. The reason is that it was meant for casual logging (as done while debugging), instead of serious _I/O_, and I felt that forcing people to use `IO` while logging would have made the process too tedious.
This chapter has been mostly theoretical, as I had to make the case for why explicit I/O is a valuable tool before I could actually explain what it was.
Here, you have learned how to walk outside and yell.
Next, I shall teach you how to walk inside and whisper... to the host platform.
See you in the next chapter!
Chapter 13: JVM inter-operation
Where you will cross the great divide.
No language is an island, and even compiled-to-native languages need to have some FFI (or foreign function interface) to interact with C, Fortran or some other language.
There's a ton of awesome infrastructure out there that was implemented in other technologies, and there is no reason for us not to take a piece of that pie.
The beautiful thing about the inter-operation mechanism offered by Lux is that it can be used to interact with any language running on the JVM (not only Java).
Although, due to its simplicity, it's better suited for interacting with programs originally written in Java, as other languages tend to implement some tricks that make the classes they generate a little bit... _funky_.
So, what do I need to work with the JVM?
Basically, just 3 things:
- The means to consume the resources it provides (i.e. classes, methods, fields, and objects).
- The means to create your own resources (i.e. class definitions).
- The means to access its special features (such as synchronization).
Let's explore them.
By the way, the only module relevant to this chapter is `library/lux/ffi`.
Importing classes, methods, and fields
It's all done with the help of the import macro:
... Allows importing JVM classes, and using them as types.
... Their methods, fields and enum options can also be imported.
(import java/lang/Object
["[1]::[0]"
(new [])
(equals [java/lang/Object] boolean)
(wait [int] "io" "try" void)])
... Special options can also be given for the return values.
... "?" means that the values will be returned inside a Maybe type. That way, null becomes .#None.
... "try" means that the computation might throw an exception, and the return value will be wrapped inside the Try type.
... "io" means the computation has side effects, and will be wrapped inside the IO type.
... These options must show up in the following order ["io" "try" "?"] (although, each option can be used independently).
(import java/lang/String
["[1]::[0]"
(new [[byte]])
("static" valueOf [char] java/lang/String)
("static" valueOf "as" int_valueOf [int] java/lang/String)])
(import (java/util/List e)
["[1]::[0]"
(size [] int)
(get [int] e)])
(import (java/util/ArrayList a)
["[1]::[0]"
([T] toArray [[T]] [T])])
... The class-type that is generated is of the fully-qualified name.
... This avoids a clash between the java.util.List type, and Lux's own List type.
... All enum options to be imported must be specified.
(import java/lang/Character$UnicodeScript
["[1]::[0]"
("enum" ARABIC CYRILLIC LATIN)])
... It should also be noted, the types that show up in method arguments or return values may only be Java classes, arrays, primitives, void or type-vars.
... Lux types, such as Maybe cannot be used (otherwise, they'd be confused for Java classes).
(import (lux/concurrency/async/JvmAsync A)
["[1]::[0]"
(resolve [A] boolean)
(poll [] A)
(wasResolved [] boolean)
(waitOn [lux/Function] void)
("static" [A] make [A] (lux/concurrency/async/JvmAsync A))])
... Also, the names of the imported members will look like Class::member
(java/lang/Object::new [])
(java/lang/Object::equals [other_object] my_object)
(java/util/List::size [] my_list)
(java/lang/Character$UnicodeScript::LATIN)
This will be the tool you use the most when working with the JVM.
As you have noticed, it works by creating functions for all the class members you need.
It also creates Lux type definitions matching the classes you import, so that you may easily refer to them when you write your own types later in regular Lux code.
It must be noted that `import` requires that you only import methods and fields from their original declaring classes/interfaces.
What that means is that if class `A` declares/defines method `foo`, and class `B` extends `A`; to import `foo`, you must do it by importing it from `A`, instead of `B`.
Writing classes
Normally, you'd use the class macro:
... Allows defining JVM classes in Lux code.
... For example:
(class "final" (TestClass A) [Runnable]
... Fields
("private" foo boolean)
("private" bar A)
("private" baz java/lang/Object)
... Methods
("public" [] (new [value A]) []
(exec
(set foo #1)
(set bar value)
(set baz "")
[]))
("public" (virtual) java/lang/Object
"")
("public" "static" (static) java/lang/Object
"")
(Runnable [] (run) void
[])
)
... The tuple corresponds to parent interfaces.
... An optional super-class can be specified before the tuple. If not specified, java.lang.Object will be assumed.
... Fields and methods defined in the class can be used with special syntax.
... For example:
... (get resolved), for accessing the 'resolved' field.
... (set resolved #1) for modifying it.
... (new! []) for calling the class's constructor.
... (resolve! container [value]) for calling the 'resolve' method.
And, for anonymous classes, you'd use object:
... Allows defining anonymous classes.
... The 1st tuple corresponds to class-level type-variables.
... The 2nd tuple corresponds to parent interfaces.
... The 3rd tuple corresponds to arguments to the super class constructor.
... An optional super-class can be specified before the 1st tuple. If not specified, java.lang.Object will be assumed.
(object [] [Runnable]
[]
(Runnable [] (run self) void
(exec
(do_something some_value)
[])))
Special features
- Accessing class objects.
... Loads the class as a java.lang.Class object.
(class_for java/lang/String)
- Test instances.
... Checks whether an object is an instance of a particular class.
... Caveat emptor: Can't check for polymorphism, so avoid using parameterized classes.
(when (check java/lang/String "YOLO")
{.#Some value_as_string}
{.#None})
- Synchronizing threads.
... Evaluates body, while holding a lock on a given object.
(synchronized object-to-be-locked
(exec
(do something)
(do something else)
(finish the computation)))
Calling multiple methods consecutively
... Call a variety of methods on an object. Then, return the object.
(to object
(ClassName::method1 arg0 arg1 arg2)
(ClassName::method2 arg3 arg4 arg5))
`to` is inspired by Clojure's own `doto` macro.
The difference is that, whereas Clojure's version pipes the object as the first argument to the method, Lux's pipes it at the end (which is where method functions take their object values).
The library/lux/ffi module offers much more, but you'll have to discover it yourself by heading over to the documentation for the Standard Library.
Host platform inter-operation is a feature whose value can never be understated, for there are many important features that could never be implemented without the means provided by it.
We're actually going to explore one such feature in the next chapter, when we abandon our old notions of sequential program execution and explore the curious world of concurrency.
See you in the next chapter!
Chapter 14: Concurrency
Where you will harness the power of modern computing.
Concurrency is one of the most important subjects in modern programming because of the pressing need to improve the efficiency of programs; coupled with the troublesome limitation that increasing the speed of individual CPUs is becoming harder and harder.
The solution, at the moment, is to increase the number of CPUs in the machine to allow programmers to run their code in parallel.
The problem is that new models of computation which take concurrency into account have had to be developed to address this need, but nobody knows yet which of the many alternatives is the right one, or if there is even a right one at all.
Until now, most programmers just thought about their programs in a purely sequential manner; and, as we explore some of these concurrency models, you'll notice that they try to restore some of this feeling, while at the same time scheduling work to happen concurrently, without (almost) any programmer involvement.
Lux takes the approach of providing the means to use multiple concurrency models, since I couldn't decide on which was the right one.
Some languages (like Erlang and Go) choose to commit themselves to one model or another, but I'm not confident that the software industry (as a whole) is experienced enough with concurrency as to declare any one model the winner.
The result: more variety.
And watch out, because the amount of concurrency models may increase with future Lux releases.
Anyhow, let's quit the chit-chat and dive in!
Asynchronous computations
This is my favorite one, because it can be used for almost anything, whereas I see the other modules as more specialized tools for certain use cases.
This model is based on concepts you may be familiar with: futures and promises.
Futures are basically concurrent and asynchronous computations which run and yield a value, which you can later access.
Promises are more like storage locations (which may be set only once), which you can use for communication by setting their values in one process, and reading it from another.
Some languages offer both (often used in conjunction), while others only offer one (while also kind of giving it properties of the other).
I pretty much came to the conclusion that, for all intents and purposes, their similarities were much greater than their differences.
So, I just fused them.
And so, Lux implements futures/promises in the library/lux/control/concurrency/async module, by means of the Async type.
You can run IO computations concurrently using the future function (which returns an Async that will contain the result of the computation).
You can also bind functions to Async values in order to be notified when they have been resolved (the term for when the value of the Async is set).
By means of this ability to watch Async values, it's possible to implement Functor and Monad for Async, which is precisely what is done in the standard library.
The result is that, through the do macro, you can implement complex concurrent and asynchronous computations that look and feel just like synchronous ones.
If you're curious about how that looks, take a peek:
(def .public (and left right)
(All (_ a b) (-> (Async a) (Async b) (Async [a b])))
(do monad
[a left
b right]
(in [a b])))
Oh, and did I mention there are _combinators_ in that module?
If you didn't know there was some magic going on in the Async type, you wouldn't have suspected this was concurrent code.
It looks just like any other old synchronous code you might have use with any other monad.
Pretty neat, huh?
Functional Reactive Programming
FRP is based on the idea of values that change over time, and structuring your programs to dynamically respond to those changes in a reactive way.
The way its implemented in Lux is through the Channel type in library/lux/control/concurrency/frp (itself implemented on top of Async).
Channel instances are (potentially infinite) sequences of values that you can process in various ways as new values come into being.
Channel instances can be closed, but they may also go on forever if you'd like them to.
The library/lux/control/concurrency/frp module offers various functions for processing channels in various ways (some of them generating new channels), and the Channel type also happens to be a monad, so you can write fairly complex and powerful code with it.
Software Transactional Memory
Implemented in the `library/lux/control/concurrency/stm` module.
STM is quite a different beast from the other 2 approaches, in that they address the problem of how do I propagate information within the system, while STM deals with how to keep data in one place, where it can be accessed and modified concurrently by multiple processes.
It works by having variables which may be read from and written to, but only within transactions, which could be seen as descriptions of changes to be made to one (or more) variables in an atomic, consistent and isolated way.
Let's break down those last 3 terms:
- Atomic: This just means that if more than one change needs to be made in a transaction, either all gets done, or none. There is no room for partial results.
- Consistent: This just means that transactional computations will take the set of variables they operate from one valid state to another. This is largely a consecuence of transactions being atomic.
- Isolated: This means that transactions run in isolation (or, without interference) from one another, thereby ensuring no transaction may see or modify any in-trasaction value being computed somewhere else, and they all get the impression that they are the only transaction running at any given time.
For those of you familiar with relational databases, this might remind you of their ACID properties (with the caveat that Lux's STM is non-durable, as it works entirely in memory).
The way it works is by running multiple transactions concurrently, and then committing their results to the affected variables.
If 2 transactions modify any common variables, the first one to commit wins, and the second one would be re-calculated to take into account the changes to those variables.
This implies that transactions are sensitive to some "version" of the variables they involve.
That is the mechanism used to avoid collisions and ensure no inconsistencies ever arise.
The relevant types are Var, which corresponds to the variables, and STM which are computations which transform transactions in some way and yield results.
Like IO and unlike Async, just writing STM computations doesn't actually run them, and you must call the commit! function to actually schedule the system to execute them (receiving an Async value for the result of the transaction).
You may also follow! variables to get Channels of their values if you're interesting in tracking them.
The Actor Model
The actor model is also very different from the other models in that, while they deal with computations which produce values concurrently, the actor model is all about processes running concurrently and communicating with one another.
You can't run an actor and just wait for it to finish to get the result. For all you know, it may never end and just run forever.
Also, interaction with actors is based on message-passing, and an actor may consume an indefinite number of such messages (and send messages to other actors).
The relevant module is the library/lux/control/concurrency/actor module, and the relevant type is:
(primitive .public (Actor s)
...
)
Actors have mailboxes for receiving messages, to which they react.
It's also possible to kill an actor (although it can also die naturally if it encounters a failure condition during normal execution).
And if it dies, you'll receive its state at the time of death, a list of unconsumed messages from its mailbox and an error message detailing the cause of death.
Just from this definition, it's easy to see that actors are stateful (a necessity for modeling a variety of complex behaviors).
To create an actor, you must first specify its Behavior:
(type .public (Behavior s)
(-> (Mail s) s (Actor s) (Async (Try s))))
This function knows how to react to incoming mail.
You can then call the spawn! function with an initial state and a Behavior.
But writing complex actors with multiple options for its messages can be messy with these tools, so a macro was made to simplify that.
(def counter
(/.Behavior Nat)
(function (_ message state self)
(message state self)))
(def (count! increment)
(-> Nat (/.Message Nat Nat))
(function (_ state self)
(let [state' (n.+ increment state)]
(async#in {try.#Success [state' state']}))))
For every message type you define, a function will be defined in your module with the same name, and taking the same arguments, plus the actor.
That function will always take the actor itself as its last argument, and will return an Async of the return type.
You can either die with a librarylux/control/try.#Failure value, or continue on to the next message with a librarylux/control/try.#Success containing an updated actor state, and a return value for the method.
The type of the return value must match the type following the method signature.
In this chapter, you have learned how to use the many tools Lux offers to tap into the multi-processor power of modern computing systems.
But if you think about it, being able to hold onto values or pass them around concurrently is rather useless unless you have some important and complex data to move around in the first place; and so far we have only dealt with fairly simple data-structures.
Well, read the next chapter if you want to learn how to take your data to the next level with the help of persistent data structures.
See you in the next chapter!
Chapter 15: Persistent data structures
Where you will learn a new way to organize your data.
So far, you know how to use variants, tuples and records to structure your data.
You're also familiar with lists, and you might even have come up with clever ways to use lists to implement key-value structures, like property lists.
But the reality is that there are many different ways to organize data, and how you implement the mechanisms to do so will have an impact on the performance of your programs.
That is why there are so many different types of data-structures, and so many different implementations for them.
But not all such implementations fit into the functional paradigm of keeping all your data immutable, and most implementations of data-structures are actually mutable, and meant for imperative programming.
Now, let's not be naïve.
Everybody can figure out that making a data-structure immutable and just copying the whole thing every time you want to make a change would make those data-structures prohibitively expensive to use.
Luckily for us, there is a way to have immutable data-structures that have reasonable performance.
The reason why they are fast enough to be used, is that they are designed to re-use as many nodes as they can whenever an update needs to be done, in order to avoid wasteful re-work wherever possible.
Make no mistake, they are still not as fast as their mutable counterparts (which you can still access by doing host-interop), but they are designed with high-performance in mind, and so they tend to be fast enough for most use-cases.
Lux offers a variety of these persistent data-structures.
Here are some examples:
Sequences
Located in `library/lux/data/collection/sequence`.
These are similar to lists in that they are sequential data-structures, but there are a few differences:
- Whereas lists prepend values to the front, sequences append them to the back.
- Random access on lists has a complexity of O(N), whereas it's O(log N) for sequences.
Sequences are a great substitute for lists whenever random access is a must, and their implementation ensures updates are as cheap as possible.
Queues
Located in `library/lux/data/collection/queue`.
Queues are the classic first-in first-out (FIFO) data-structure.
Use them whenever processing order matters, but you need to add to the back (unlike lists, where order matters but add to the front).
Dictionaries
Located in `library/lux/data/collection/dictionary`.
This is your standard key-value data-structure.
Known by other names (tables, maps, etc), dictionaries give you efficient access and updating functionality.
All you need to do is give it a Hash implementation (from library/lux/abstract/hash) for your "key" type, and you're good to go.
Sets
Located in `library/lux/data/collection/set`.
This is similar to dictionaries in that a Hash implementation is needed, but instead of it being a key-value data-structure, it only stores values (and then tells you if any given value is a member of the set).
This is a useful data-structure for modelling group membership and keeping track of things. Plus, there are several set-theoretic operations defined in that module.
Persistent data structures and Software Transactional Memory
This is a killer combination.
Instead of using mutable data-structures for your changing program data, you can just use persistent data-structures, with the mutability being delegated the the STM system.
This will make working concurrently with these data-structures a piece of cake, since you never have to worry about synchronizing/locking anything to avoid simultaneous updating, or any of the other crazy things programmers have to do to avoid data corruption.
Arrays: the not-so-persistent data structures
Located in `library/lux/data/collection/array`.
The library/lux/data/collection/array module features mutable arrays you can use if you need fast random access and mutation and are willing to run the risks involved with using mutable data.
Another possible use is to implement other data-structures (and, as it turns out, sequences and dictionaries both rely on arrays for their implementations).
Also, it's worth nothing that in the JVM, this corresponds to object arrays.
If you want primitive arrays, you should check out the functionality for that in library/lux/ffi.
It may seem odd that this chapter doesn't feature any code samples, but most of what you need to know is already located in the standard library documentation.
These data-structures are very easy to use and offer decent performance, so you're encouraged to use them to model all your data processing code.
The next chapter is going to be slightly different, in that we're going to be learning not how to write programs, but how to test them.
See you in the next chapter!
Chapter 16: Testing
Where you will learn how to avoid annoying bug reports.
Automated testing is a fundamental aspect of modern software development.
Long gone are the days of manual, ad-hoc testing.
With modern testing tools and frameworks, it's somewhat easy to increase the quality of programs by implementing comprehensive test suites that can cover large percentages of a program's functionality and behavior.
Lux doesn't stay behind and includes a testing module as part of its standard library.
The library/lux/test module contains the machinery you need to write unit-testing suites for your programs.
Not only that, but the Aedifex build tool for Lux also includes a command for testing: lux test
How do you set that up?
Let's take a look at the project.lux file for the Lux standard library itself.
[""
["identity" ["com.github.luxlang" "stdlib" "0.7.0"]
"deploy_repositories" ["snapshots" "https://oss.sonatype.org/content/repositories/snapshots/"
"releases" "https://oss.sonatype.org/service/local/staging/deploy/maven2/"]
"repositories" ["https://oss.sonatype.org/content/repositories/snapshots/"
"https://oss.sonatype.org/service/local/staging/deploy/maven2/"]]
"jvm"
["lux" ["com.github.luxlang" "lux-jvm" "0.7.0" "jar"]]
"bibliotheca"
["info" ["description" "Standard library for the Lux programming language."]
"test" test/lux._]
]
The "test" parameter specifies the name of a Lux module that serves as the entry point for testing.
Here is a summary of the file:
(.require
[library
["/" lux (.except)
[program (.only program)]
[test
["_" property (.only Test)]]
[control
["[0]" io]]
...
]])
...
...
...
(def _
(program args
(io.io (_.run! (_.times 100 ..test)))))
A test suit consists of a Test (or a composite of as many Tests as you want), which is then run!.
The times combinator allows you to execute Tests several times.
This can be very useful when using random data generation within your tests, as each run of the tests will lead to the generation of different sorts of data.
This will help you cover many possible scenarios within the same test run, and perhaps uncover tricky corner cases you wouldn't have thought of.
But where do those tests come from?
Nothing is being defined here.
Let's take a look at the tests defined in a simpler module.
Well, the run macro, from library/lux/test pulls in all the tests from the imported modules to run them later once the program starts.
To know how tests work, let's take a look at one of those modules.
From `test/lux/data/collection/stack`.
(.require
[library
[lux (.except)
[test
["_" property (.only Test)]]
[abstract
[monad (.only do)]
[\\specification
["$[0]" equivalence]
["$[0]" functor (.only Injection)]]]
[control
["[0]" maybe]]
[data
["[0]" bit (.use "[1]#[0]" equivalence)]]
[math
["[0]" random]
[number
["n" nat]]]]]
[\\library
["[0]" /]])
(def (injection value)
(Injection /.Stack)
(/.top value /.empty))
(def .public test
Test
(<| (_.covering /._)
(_.for [/.Stack])
(do random.monad
[size (at random.monad map (n.% 100) random.nat)
sample (random.stack size random.nat)
expected_top random.nat]
($_ _.and
(_.for [/.equivalence]
($equivalence.spec (/.equivalence n.equivalence) (random.stack size random.nat)))
(_.for [/.functor]
($functor.spec ..injection /.equivalence /.functor))
(_.coverage [/.size]
(n.= size (/.size sample)))
(_.coverage [/.empty?]
(bit#= (n.= 0 (/.size sample))
(/.empty? sample)))
(_.coverage [/.empty]
(/.empty? /.empty))
(_.coverage [/.value]
(when (/.value sample)
{.#None}
(/.empty? sample)
{.#Some _}
(not (/.empty? sample))))
(_.coverage [/.next]
(when (/.next sample)
{.#None}
(/.empty? sample)
{.#Some [top remaining]}
(at (/.equivalence n.equivalence) =
sample
(/.top top remaining))))
(_.coverage [/.top]
(when (/.next (/.top expected_top sample))
{.#Some [actual_top actual_sample]}
(and (same? expected_top actual_top)
(same? sample actual_sample))
{.#None}
false))
))))
There's a lot going on here.
First of all, by using the covering macro, you can tell the test suit to track the coverage that your test suite has of a given module.
That way, if your tests miss some exported/public definitions, the report you'll get after running the tests will tell you, so you can judiciously choose to either expand your coverage, or skip covering them.
The for and coverage macros then signal whenever one or more definitions are being covered by a given test.
Lux also defines some specifications, which are basically parameterizable tests, which implement consistent testing for various interfaces in the standard library.
That way, when testing an implementation of those interfaces, instead of having to copy-paste, or re-invent the testing every time, the specification is imported.
This enables consistent testing of implementations.
and allows you to sequentially compose Tests into a larger Test.
You can also see an example of how to use randomness to generate sample data for testing.
If you want to learn more about how to write tests, feel free to check out the test-suite for the Lux standard library.
It's very comprehensive and filled with good examples.
Without tests, the reliability of programs becomes a matter of faith, not engineering.
Automated tests can be integrated into processes of continuous delivery and integration to increase the confidence of individuals and teams that real value is being delivered, and that the customer won't be dissatisfied by buggy software.
Now that you know how to test your programs, you know everything you need to know to be a Lux programmer... on the JVM.
However, Lux has been expanded with support for other platforms, and it's time for you to learn about some of its capabilities.
See you in the next chapter!
Chapter 17: Cross-platform Lux
Where you will sail to exotic foreign platforms aboard the S.S. Lux.
It was always my desire for Lux to be a language that could be used to write software for multiple platforms.
I've always found it annoying to write a piece of software in one language, and then if it became necessary to run the software under different circumstances, a rewrite had to be done because the language in which the software was written was not capable of adapting to the new requirements.
In theory, programming languages are universal.
- Logic is logic, no matter the language in which it is expressed.
- Instructions are instructions, no matter who executes them.
And yet, in practice, you need JavaScript for the browser, and Swift for the IPhone (to give you some examples).
Granted, with the advent of WebAssembly, it has now become possible to have any options you want for the browser, instead of being locked to JavaScript and any languages that transpile to it.
But there is still another type of constraint that is probably not going away any time soon.
If a programmer or a company write a piece of software in some language, they have made a very significant investment.
If tomorrow, they realize that another language might be a better fit for what they need, it might be too costly for them to rewrite their software in a new language.
Cross-platform support for Lux is not just about accessing as many devices as possible, but also about being able to inter-operate with as many languages as possible.
Being able to interact with and extend a Python server, or a Lua script.
Ideally, I would like Lux to be able to inter-operate with every programming language that exists, thereby giving Lux programmers the capacity to write any program they want for any device; and the capacity to extend any codebase without being forcefully tied to any legacy language or implementation.
Sadly, the ideal world can only exist within our dreams, but (as of v0.6) steps have been taken to bring the dream closer to reality.
Currently, it is possible to write Lux programs that compile to the following target platforms:
- Java virtual machines
- JavaScript interpreters (such as browsers, and Node JS)
- Python interpreters
- Lua interpreters
- Ruby interpreters
Not only is that possible, but great care has been taken to make sure Lux works consistently across each of those platforms.
That means 2 things:
- If your Lux program behaves one way in one platform, you can expect it to behave the same way in any other, minus performance considerations.
- There is only one language for programming in any of those platforms: Lux. There is no LuxScript or PyLux. Lux is a single programming language, not a family of languages.
The 2nd point is important, because another possible design could have been to have slightly different variations of Lux as different programming languages (albeit, sharing most of their syntax and semantics), each one targeting one of those platforms.
This could have been a very nice approach, as each of those platforms works slightly differently, so it's not crazy to think that each variation of Lux could adapt itself to those differences to offer a programmign experience closer to the platform.
The problem with this approach is that libraries written for one platform might not work for the others (or work correctly) because they would make certain assumptions about how things work that would be true for their platform of origin, but not in general.
This approach would allow Lux programmers to target different platforms, but it would make sharing code between them impossible at worst, and risky at best.
Instead, I've designed the semantics and the feature set of Lux to be independent of any host platform.
When the feature-set of a platform fits well with the feature-set of Lux, I use the platform's resources to implement Lux's functionality.
And when Lux needs something the platform does not offer, but does not disallow either, I emulate the feature in other to ensure a consistent experience.
This means all of Lux's naturals and integers are 64-bit, on the JVM (which supports them), and even on JavaScript (where they are emulated).
And this also means that Lux's concurrency mechanisms work as expected on the JVM (which allows multi-core processing), and on JavaScript/Python/Lua/Ruby (which don't).
In order to compile a Lux program to any of these alternative platforms, you must use a different compiler for each.
An Aedifex project.lux file allows for a "lux" option to specify (as a dependency) the (Lux) compiler you wish to use.
This option can be omitted, in which case it will pick, as a default value: ["com.github.luxlang" "lux-jvm" "0.7.0" "jar"].
Here are the compilers for the alternative platforms:
- For JavaScript:
["com.github.luxlang" "lux-js" "0.7.0" "js"] - For Python:
["com.github.luxlang" "lux-python" "0.7.0" "jar"] - For Lua:
["com.github.luxlang" "lux-lua" "0.7.0" "jar"] - For Ruby:
["com.github.luxlang" "lux-ruby" "0.7.0" "jar"]
You don't need to use any special command on Aedifex in order to compile Lux to any alternative platform.
Just set the compiler, and build/test your program as usual.
For a thorough specification of what Aedifex can do, please refer to [Appendix H](appendix_h.md).
In the same way that the JVM compiler produces a single executable JAR file, each of these compilers will produce a single executable .js/.py/.lua/.rb file that can directly be executed with the usual interpreters for those languages.
You might be wondering, though, how is it possible to share Lux code that is meant to work on these different platforms, given that they have different features, and different libraries.
How is it possible to write Lux code that works everywhere, instead of being tied to the minutiae of each platform?
Lux offers 2 different mechanisms to write cross-platform code.
One of them is meant to be used when most of the code you're writing is the same regardless of platform, and you just need to add some snippets here and there to access some specific things in each platform.
And the other is meant for when the changes are so massive, you might as well just write different files for different platforms.
First, let's go with the smaller mechanism:
(def js "JavaScript")
(for ["JVM" (do jvm stuff)
..js (do js stuff)]
(do default stuff))
The for macro allows you to specify the code to use for each platform.
Each Lux compiler has a name (as a Text) for the platform it compiles to, and this information is made available to macros through the state of the compiler.
The for macro just compares this name to whatever options you give it in order to select what code to use when compiling to a given platform.
Additionally, it is possible to provide an (optional) snippet of code, to use in case there isn't a more specific snippet available.
Also, as you can see, the names for the platforms don't need to be provided as Text literals, as for knows how to resolve the names of definitions to get the options that way.
This is specially useful to avoid typos causing trouble when using for.
The module library/lux/target contains constants with the names of currently supported platforms, so it's a good idea to rely on that instead of manually specifying the names.
If you check out `library/lux/target`, you might notice that there are more constants than there are currently supported platforms.
This is because I made the effort to add additional platforms to Lux, but due to complications, I had to postpone them for later.
The constants were left there anyway, since there is some code that depends on them on some of the more platform-specific parts of the standard library.
As I finish the remaining back-ends, these constants will gain relevance again.
To give you an example of for in action, here is a definition from the library/lux/data/text module:
(def .public (replaced pattern replacement template)
(-> Text Text Text Text)
(for [@.old
(as Text
("jvm invokevirtual:java.lang.String:replace:java.lang.CharSequence,java.lang.CharSequence"
(as (Primitive "java.lang.String") template)
(as (Primitive "java.lang.CharSequence") pattern)
(as (Primitive "java.lang.CharSequence") replacement)))
@.jvm
(as Text
("jvm member invoke virtual" [] "java.lang.String" "replace" []
(as (Primitive "java.lang.String") template)
["Ljava/lang/CharSequence;" (as (Primitive "java.lang.CharSequence") pattern)]
["Ljava/lang/CharSequence;" (as (Primitive "java.lang.CharSequence") replacement)]))
... TODO: Comment/turn-off when generating a JS compiler using a JVM-based compiler because Nashorn's implementation of "replaceAll" is incorrect.
@.js
(as Text
("js object do" "replaceAll" template [pattern replacement]))
@.python
(as Text
("python object do" "replace" template pattern replacement))
... TODO @.lua
@.ruby
(as Text
("ruby object do" "gsub" template pattern replacement))
@.php
(as Text
("php apply" (expected ("php constant" "str_replace"))
pattern replacement template))
... TODO @.scheme
... TODO @.common_lisp
... TODO @.r
]
... Inefficient default
(loop (again [left ""
right template])
(when (..split_by pattern right)
{.#Some [pre post]}
(again ($_ "lux text concat" left pre replacement) post)
{.#None}
("lux text concat" left right)))))
This function implements text-replacement in a generic way, while also taking advantage of platform-specific functionality where available.
The 2nd mechanism for writing cross-platform code is to specify platform-specific Lux files.
The way this works is by adding a secondary extension to your Lux files.
A normal Lux file looks like this: foo.lux.
When a Lux compiler sees that, it assumes the file contains code which is expected to work on any platform Lux can compile to.
However, it is possible to specify that a file contains code that is only meant for a specific platform, like this:
- For the JVM:
foo.jvm.lux - For JavaScript:
foo.js.lux - For Python:
foo.py.lux - For Lua:
foo.lua.lux - For Ruby:
foo.rb.lux
If you're using, let's say, the JavaScript compiler for Lux (i.e. ["com.github.luxlang" "lux-js" "0.7.0" "js"]), whenever you import a module as a dependency, the compiler will first look for a file with the .js.lux extension, and if it fails to find one, it will look for a file with the plain .lux extension.
What happens if I do not have a .js.lux file, but I do have files with the other special extensions?
A Lux compiler will always ignore files with extensions for platforms other than its own.
It will only ever take into account its own custom extension, and the general extension, so you don't need to worry about the wrong code being compiled.
A good example of this mechanism in action is the library/lux/ffi module.
This module provides several macros for making host-platform interoperation very easy to do.
Since the whole point of library/lux/ffi is platform interoperation, there is no point in there ever being a generic library/lux/ffi.lux file.
Instead, there is a library/lux/ffi.jvm.lux file, a library/lux/ffi.js.lux file, a library/lux/ffi.py.lux file, a library/lux/ffi.lua.lux file, and a library/lux/ffi.rb.lux file.
It makes the most sense to provide this module as a set of platform-specific files, instead of a single file that uses for, because everything in those files is platform-specific and there can be no meaningful re-use, so a version of that module which is a single file using for, would be an incomprehensible monstrosity.
But, by splitting the code into platform specific files, everything can be kept nice and tidy.
You might also want to take a close look at the documentation for `library/lux/ffi` to see what macros are available.
I wouldn't be surprised if you looked at the previous example of the `replaced` function and thought: _YUCK!_.
Don't worry about it. Such ugly code is un-characteristic of _the Lux experience_.
The reason why that code looks that way is because the `library/lux/data/text` gets implemented before `library/lux/ffi`, and so it cannot use any of the machinery specified therein.
_Your code_ will have access to `library/lux/ffi`, so you can write much nicer code that doesn't have to concern itself with the low-level nitty-gritty details.
You might be puzzled by what you saw in that replaced example.
You're calling text as if it was a function?
Not quite.
You see, not all functionality a programming language provides can be implemented entirely within the programming language.
Sometimes, there are primitive bits of functionality that have to be baked into the language from the get-go.
Lux's mechanism for exposing those bits is as extensions to the compiler.
Some of these extensions are common to each compiler, and can be expected to be around regardless of whether you're compiling to the JVM, to JavaScript, or anywhere else.
Other extensions are host-specific, and are only meant to be around for a specific platform.
Either way, Lux uses the same mechanism for all of them: the humble extension.
You want to know what's the coolest thing about extensions?
You can write your own, and by doing so you can teach the compiler how to type-check, optimize and even generate code for your own new types of expressions.
Sounds cool?
See you in the next chapter!
Chapter 18: Extensions
Where you will teach new tricks to an old compiler.
Lux is a member of the Lisp family of languages.
As such, it has macros, which are a classic staple of lisps which allow programmers to extend the syntax of their language to implement all sorts of new features, domain-specific-languages (or DSLs, for short), optimizations, and even entire paradigms, such as object-oriented programming.
However, compilers do much more than just process syntax.
In statically-typed languages, compilers also type-check code; and even in dynamically-typed languages, compilers often optimize your code, and generate output code for either the machine the program will run on, or some sort of virtual machine or interpreter.
Macros, however, don't extend any of these other aspects of compilation; only the syntax of the language.
And so, even lisps keep the programmer out of most of the activities of the compiler.
Lisps may be more extensible than most other languages, but they are not completely extensible.
With Lux, however, I want to take the traditional extensibility of lisps and take it to its ultimate expression.
Not only should syntax be extensible, but also type-checking, optimization, and code-generation.
And who knows? Maybe even more aspects will be extensible in the future.
But, for now, let's see what Lux has got to offer.
Now, before we get into the details, let's first discuss the syntax being used for just a moment.
("js object do" "replaceAll" template [pattern replacement])
What is up with that text-based syntax?
Well, here's the deal.
Most lisps have what they call special forms.
Basically, these are macro-like expressions which provide some fundamental features of the language.
Special forms, however, are not macros.
They are default types of expressions provided by the compiler/interpreter.
They are named by symbols/identifiers, such as let, or if, and because of that, the symbols/identifiers used to name them are reserved.
That is to say, if there is a special form named if, that means you cannot make your own definition named if, because then the compiler/interpreter might get confused as to whether somebody is using the special form, or an arbitrary definition.
The term keyword is often used in programming languages to refer to symbols/identifiers considered to be special by the compiler/interpreter, and reserved for the exclusive use of the language designer.
Personally, I hate the idea of having special privileges that users of Lux lack.
I think that, as a designer, I'd be cheating if I could do things nobody else could, such as using reserved names.
So, I didn't like the idea of naming extensions with identifiers, because then those identifiers would effectively become reserved keywords.
Instead, since an expression that attempts to call Text as a function is meaningless in Lux, as Texts are not functions; I thought it would be nice to then use that otherwise meaningless syntax by having Text literals name extensions.
So, that's the story.
Extensions are named by Text literals to avoid reserving identifiers as keywords, thereby protecting the programmer's freedom to name their definitions however they want.
Now that we've got the origin story out of the way, let's talk business.
How can you install your own extensions?
Well, the first thing to note is that there are different types of extensions.
Remember what I said about the compiler doing different activities, such as type-checking, optimization and code-generation?
Well, you can't really expect an extension meant for type-checking to work during code-generation.
The work done on either phase would be too different to what was necessary for the other phase.
And so, Lux provides 4 different types of extensions.
However, as far as this tutorial is concerned, we will only be covering 3 of them, as the 4th one is very different to the other 3 in what its purpose is and what you're supposed to do with it, and I'd feel more comfortable teaching it later, once I figure out a reasonable way for normal Lux programmers to make use of it.
The first type of extension we'll see is the Analysis extension:
(.require
[library
[lux (.only)
[extension (.only analysis synthesis generation)]
[abstract
["[0]" monad {"+" [do]}]]
[control
["<>" parser
["<[0]>" code]
["<[0]>" analysis]
["<[0]>" synthesis]]]
[data
[collection
["[0]" sequence]]]
[meta
["@" target
["[0]" jvm]
["[0]" js]
["[0]" python]
["[0]" lua]
["[0]" ruby]]
[tool
[compiler
["[0]" phase]
[language
[lux
["[0]" analysis]
["[0]" synthesis]
["[0]" directive]
[phase
[analysis
["[0]" type]]]]]]]]]])
(analysis ("my triple" self phase archive [elementC <code>.any])
(do phase.monad
[[type elementA] (type.with_inference
(phase archive elementC))
_ (type.infer (.Tuple type type type))]
(in (analysis.tuple (list elementA elementA elementA)))))
If you want to write your own extensions, the first you'll want to do is import the library/lux/extension module.
It contains macros for easily implementing extensions.
These macros handle parsing the inputs to your extensions using the monadic parsing infrastructure Lux provides.
Each type of extension takes a different type of input, and produces a different type of output.
In the case of Analysis extensions, they take (List Code) as an input, and produce a single Analysis node as output.
By the way, _"analysis"_ is the name Lux gives to the process of type-checking and verifying program correctness.
Here, we've got a trivial extension where we take a single value, and we produce a triple of the same value.
As you can see, we're doing some type-inferencing to first figure out the type of our input value, and then to signal to the compiler what the type of our ("my triple" ???) ought to be.
Also, one thing all phases of the compiler (such as analysis) have in common is that they are instances of the `Phase` type, defined in `library/lux/tool/compiler/phase`.
That is why I make use of the `Monad` implementation for `Phase` in this example.
Now, we have converted fairly exceptional code, using our "my triple" extension, into fairly normal code that Lux knows how to handle, which is just a 3-tuple of the same value.
As you can probably guess, this great power comes with great responsibility.
Since you are in total control of the type-checking that is happening, it is entirely possible to fool the compiler into believing the wrong things about your code.
It is very important to be careful, when implementing extensions, that the output of those extensions is correct in every situation; because, unlike with normal Lux code, the compiler cannot verify that your extension is not doing something that it shouldn't.
I have gifted you promethean fire.
DO NOT MAKE ME REGRET IT ;)
Also, you might have noticed that, besides the input code we're parsing, our extension is receiving 3 extra parameters:
self: This is just aTextthat contains the name of the extension itself (in this case, the value"my triple"). It might seem redundant, but it can be useful to refer to the extension's name within its own implementation without having to constantly repeat the name literally.phase: This is the Lux compiler's implementation of the phase an extension is part of (in this case, theAnalysisphase). It allows you to process your inputs the same way the Lux compiler does, before doing something special with them (like triplicating them, as in this example).archive: TheArchiveis a special data-structure used by the Lux compiler to store valuable information gathered during the compilation process. It is very important, but this might not be the best place to get into its relevance and what can be done with it. For now, just make sure to pass it around when invoking thephasefunction.
(analysis ("my quadruple" self phase archive [elementC <code>.any])
(do phase.monad
[[type elementA] (type.with_inference
(phase archive elementC))
_ (type.infer (.Tuple type type type type))]
(in {analysis.#Extension self (list elementA)})))
(synthesis ("my quadruple" self phase archive [elementA <analysis>.any])
(do phase.monad
[elementS (phase archive elementA)]
(in (synthesis.tuple (list elementS elementS elementS elementS)))))
The Synthesis phase is where we do optimizations.
Synthesis extensions take a (List Analysis) as input, and produce a single Synthesis node as output.
Currently, the optimization infrastructure Lux provides is not very sophisticated, and much of it has yet to be properly exposed to programmers, so you'll probably not be working too much in this layer for now.
(analysis ("my quintuple" self phase archive [elementC <code>.any])
(do phase.monad
[[type elementA] (type.with_inference
(phase archive elementC))
_ (type.infer (.Tuple type type type type type))]
(in {analysis.#Extension self (list elementA)})))
(generation ("my quintuple" self phase archive [elementS <synthesis>.any])
(do phase.monad
[elementG (phase archive elementS)]
(in (for {@.jvm (row.row (#jvm.Embedded elementG)
(#jvm.Stack #jvm.DUP)
(#jvm.Stack #jvm.DUP)
(#jvm.Stack #jvm.DUP)
(#jvm.Stack #jvm.DUP))
@.js (js.array (list elementG
elementG
elementG
elementG
elementG))}))))
Now, let's talk about the star of the show: generation extensions.
_Generation_ is just a short way of saying _code-generation_, and it's the part of the compiler that generates the actual output that you eventually execute as a program.
Also, I'd like to use this opportunity to point out that when Lux encounters a usage of an extension during any phase, and it does not know this extension, it just proceeds to process the parameters to the extension, and then hands over the extension call, with the processed parameters, to the next phase.
As a consequence, we can write a generation extension without having to write the preceeding `Synthesis` extension, because we can trust Lux to handle things reasonably and then use our generation extension when its turn comes up.
Generation extensions take a (List Synthesis) as input, and produce suitable code as output.
Since Lux offers different compilers that target different platforms, it is impossible for this phase to produce a single type of output.
Instead, the type of the output of generation extensions will depend on which compiler you're using.
In this case, we're implementing an extension that only works on the JVM, and on JavaScript.
In the case of the JVM, our output type is library/lux/target/jvm.Bytecode; and in the case of JavaScript, our output type is library/lux/target/js.Expression.
I will not go into detail about the machinery in these modules, as that is what the documentation of the Standard Library is for.
But, suffice it to say, that there are plenty of functions and useful machinery in there to write syntactically correct output code; and this is the same machinery that the Lux compiler itself uses, so you don't need to worry about your output being in any way different from what Lux itself produces.
My goal with extensions has been to take the ideas of Lisp and take them to their ultimate conclusions.
To push what is possible as far as imagination can reach.
If you have no clue what to do with this power, you are in good company.
The tools at our disposal both enable, and limit, our imagination.
If you have been using lisp languages for a while, you're probably well aware of the immense power that macros can provide.
Sadly, most programmers are unfamiliar with macros, and many among them want to stay away from macros, because they fear that all that power will be abused and get out of hand.
They are afraid of power because they never learned how to use it.
Even we lispers have been denied the power to completely control the behavior of our compilers, and so we do not know what's possible and what amazing progress can be made with these tools.
It is my hope that now that I have exposed the means to control and extend the compiler in these new directions, brilliant minds will seize this opportunity to discover new means to extend the power of programmers.
Perhaps you, my dear reader, will be one such mind.
Before you close this book, you might want to read the last few words I've got to offer.
Conclusion
This may be a short book, but I hope it did its job of showing you the power that Lux has got to offer.
The beauty of a programming language is that it is a tool for the mind.
It gives you the structure and order you need to turn your thoughts into logical machines that consume and transform and move around information on computers.
But, the structure a programming language provides is not constraining, but enabling.
It gives you the power to perform amazingly complex tasks with relatively little effort.
My mission with Lux has been (and continues to be) to create a language that maximizes the effectiveness of programmers, by making it as easy as possible to achieve great levels of complexity, without getting buried by it.
Lux is still in its adolescence.
What you have learned is Lux version 0.7.0.
In future releases, much more power will be added to the language, more platforms will be within reach of Lux programmers, and better performance will be achieved, with little to no effort on the side of programmers.
The future of Lux is bright, and this is just the beginning of an amazing adventure for you, my dear reader.
Appendix A: Import syntax
You've already seen some import syntax, but now you'll see all the options available.
If you recall Chapter 1, there was this example code:
(.require
[library
[lux (.except)
[program (.only program)]
["[0]" debug]
[control
["[0]" io]]]])
Here, we're importing the library/lux module.
The (.except) option means locally import every definition exported by the library/lux module.
This allows usage of those definitions without having to give them the library/lux. prefix, or even the . shortcut prefix.
This may cause some issues if you import 2 definitions with the same name from different modules; or if you get a definition from one module, but then write your own definition with the same name in your code.
In those circumstances, the compiler will complain, saying that you can't re-define X; where X is the name of the definition.
Then, we import library/lux/program, but we only import locally the program definition.
That is what the .only option allows.
There is also a `.only` option which means locally import everything **except** the specified definitions_.
You could use it like this: `[your_module (.except foo bar baz)]`
Finally, we import both the library/lux/debug and library/lux/control/io modules.
In neither case do we import any of their definitions locally.
We also give both of those modules local aliases.
That is what that "[0]" syntax does.
The .require macro recognizes that syntax for aliases and replaces the [0] with the import name directly to the right.
That means:
"[0]"+debug=debug"[0]"+io=io
This might not seem like a big deal, but the aliasing syntax allows you to give imports arbitrary names, so long as you take into account the substitutions that may happen.
So, for example:
"my_[0]"+debug=my_debug"[0]_[0]"+io=io_io
It is also important to note that while imports can be nested for convenience, they don't have to be.
The .require declaration could just as easily been written like this:
(.require
[library/lux (.except)]
[library/lux/program (.only program)]
["debug" library/lux/debug]
["io" library/lux/control/io])
You might also guess that library was not imported as a module because it was neither given an alias, not had any definitions specified as local imports.
Any module-path fragments included in the import syntax without such options will not be imported and will simply be assumed to be part of the module-paths of the sub-modules specified under them.
It is also possible to have the .require macro open interface implementations for you when importing the modules that contain them.
For example:
(.require
[library
[lux (.except)
[data
[collection
["[0]" list (.use "[1]::[0]" functor monoid)]]]]])
The import above would locally import:
list::each, fromfunctor.list::identity, frommonoid.list::composite, frommonoid.
Here, we can also see some additional syntax for aliasing.
First of all, when opening implementations, aliasing syntax is used to determine the names of the local implementation imports.
The [0] is replaced with the name of the implementation member.
The [1] is bound to the name of the context of the import.
In this case, the implementations are coming from the library/lux/data/collection/list module, so that is the context.
And since that module has been imported with the local alias list, that is the name that replaces the [1] in the aliasing syntax for the implementation imports.
And that is how we end up with the list of names I enumerated above.
The [1] syntax for aliasing can also be used between modules, and not just when importing implementation members.
For example:
(.require
[library
[lux (.except)
[data
["[0]" collection (.only)
["[1]/[0]" list (.use "[1]::[0]" functor monoid)]]]]])
Would locally import:
collection/list::each, fromfunctor.collection/list::identity, frommonoid.collection/list::composite, frommonoid.
The context between module imports corresponds to the closest ancestor path which has itself been aliased.
Non-aliased paths don't count as context.
This means:
(.require
[library
[lux (.except)
["[0]" data
[collection
["[1]/[0]" list (.use "[1]::[0]" functor monoid)]]]]])
Would locally import:
-
data/list::each, fromfunctor. -
data/list::identity, frommonoid. -
data/list::composite, frommonoid.Also, that
"_"/"ignore"syntax you may have noticed means do not import this module; just give it an alias I can refer to later as a context.
I should also note that you can both locally import definitions and open implementations as parts of the same module import.
For example:
(.require
[library
[lux (.except)
[data
[collection
["[0]" list (.only repeated size) (.use "[1]::[0]" monad)]]]]])
Another important feature of module imports is relative addressing, which comes in 2 flavors.
For the first one, suppose you have the following directory structure:
program
foo
bar
baz
quux
test
foo
bar
baz
quux
And you're writing code in the program/foo/baz module.
You can import other modules in the hierarchy like this:
... In program/foo/baz
(.require
[library
[lux (.except)]]
["[0]" /quux] ... program/foo/baz/quux, aliased as /quux
["[0]" //bar] ... program/foo/bar, aliased as //bar
["[0]" ///] ... program, aliased as ///
)
A single forward slash (/) signifies "this module" in the hierarchy, so anything after the forward slash is assumed to be under this module.
Two forward slashes (//) signify "the module above", and any forward slash after that allows you to go further up the hierarchy.
In the case of program, it's enough to just specify three forward slashes (///) for the .require macro to know which module you're referring to.
You can think about it like this:
program/foo/baz+/+quux=program/foo/baz/quuxprogram/foo/baz+//+bar=program/foo+/+bar=program/foo/barprogram/foo/baz+///=program/foo+//=program+/=program
Also, this relative path syntax can be nested, like so:
... In program/foo/baz
(.require
[library
[lux (.except)]]
[/
["[0]" quux]] ... program/foo/baz/quux, aliased as quux
[//
["[0]" bar] ... program/foo/bar, aliased as bar
]
["[0]" ///] ... program, aliased as ///
)
Or even:
... In program/foo/baz
(.require
[library
[lux (.except)]]
[/
["[0]" quux] ... program/foo/baz/quux, aliased as quux
[//
["[0]" bar] ... program/foo/bar, aliased as bar
["program" //] ... program, aliased as program
]])
You may have noticed that when importing program, we went from /// to //.
That is because, since it's nested under another //, it's relative to program/foo instead of program/foo/baz, so only 1 step up is necessary instead of the 2 steps a /// would provide.
For the second way to do relative imports, you can see this example:
... In program/foo/baz
(.require
[library
[lux (.except)]]
[\\test
["[0]" /] ... test/foo/baz, aliased as /
]
... Alternatively
["[0]" \\test] ... test/foo/baz, aliased as \\test
... Or
[\\
[\test
["[0]" /] ... test/foo/baz, aliased as /
]]
)
The backslash (\) works in the reverse direction to the forward slash (/).
If the forward slash allows you append paths to the back, and to move up the hierarchy from the end; then the backslash allows you to append paths to the front, and the move down the hierarchy from the beginning.
Why would you want such a thing?
Because it allows you to easily establish parallel hierarchies of modules, which is a useful way to separate orthogonal aspects of your program (like the program and test hierarchies in our example).
Then, by using this relative syntax, you can refer to one hierarchy from another in an easy way.
Appendix B: Math
Math in Lux is a bit different from what you might be used to in other languages.
For starters, Lux is a lisp, which means that it uses prefix syntax for everything.
That means, familiar operations such as 3 + 5 or 8 = 8 get written as (+ 3 5) and (= 8 8).
There's also the issue of different operators for different types.
Whereas other programming languages often overload the math operators +, -, >, etc. for all numeric (and some non-numeric) types, Lux actual offers different versions for different types (defined in the respective modules for those types).
The differences may look trivial, but since the numeric types are treated differently in Lux, you must be aware of which function-set you're using when working with your data.
However, this is not the biggest difference in Lux's math operators in comparison to other languages.
The difference that takes the crown is the ordering of the arguments.
What do I mean?
In most languages you'd write 4 - 7.
In other lisps you'd write (- 4 7).
But in Lux, you'd write (- 7 4).
This even applies to ordering operators.
In most languages you'd write `4 < 7`.
In other lisps you'd write `(< 4 7)`.
But in Lux, you'd write `(< 7 4)`.
What is going on!? This is so bizarre!
Calm down.
Everything is going to be fine.
What's going on is that in functional programming, there is this convention of putting the most significant argument to a function as the last one.
In the case of math functions, this would be the argument on which you're operating.
I call it the "subject" of the function.
In the case of the subtraction operation, it would be the 4, since you're subtracting 7 from it.
In most lisps, the order of the arguments is such that the subject is the first argument; but not so in Lux.
Now, you may be wondering: what could possibly be the benefit of making this bizarre change?
Piping. Piping convinced me to make this change.
You see; this might look very impractical to those accustomed to the old way, but when you're writing complex calculations with many levels of nesting, being able to pipe your operations helps a lot, and this style of doing math lends itself perfectly for it.
Consider this:
(|> x (/ scale) (pow 3.0) (- shift))
If I was using the traditional way of doing math, I wouldn't be able to pipe it, and it would look like this:
(- (pow (/ x scale)
3.0)
shift)
`pow` is the _power_ function, located in `library/lux/math`.
You can complain all you want about me breaking with tradition, but that just looks ugly.
So, I'm just going to say it: I broke with tradition because tradition wasn't helping, and I'm not one to just comply because I'm supposed to.
However, I'm not without a heart; and I know that a lot of people would prefer to have a more... traditional way of doing math.
So, for you guys, I've introduced a special macro in the library/lux/math/infix module.
It's called infix, and it allows you to do infix math, with nested expressions.
Here's an example:
(infix [[3.0 pow 2.0] + [5.0 * 8.0]])
So, that corresponds to 3^2 + 5*8.
Note that infix doesn't enforce any grouping rules, since you can actually use it with arbitrary functions that you import or define (even with partially applied functions).
The rule is simple, the argument to the right of the operator will be taken first, and then the argument to the left.
So [3.0 pow 2.0] becomes (pow 2.0 3.0), and [5.0 * 8.0] becomes (* 8.0 5.0).
Thus, the infix syntax is transformed into Lux's prefix variation.
I know that Lux's way of doing math is a bit... foreign; but I made that change to ensure math fit the rest of the language perfectly.
Hopefully you'll come to see that getting used to the new way is a piece of cake and has its advantages as soon as you write complex calculations.
Appendix C: Pattern-matching macros
Pattern-matching is a native Lux feature, and yet when is a macro.
Why?, you may wonder. What does being a macro add to the mix?
Well, as it turns out, by making when be a macro, Lux can perform some compile-time calculations which ultimately enable a set of really cool features to be implemented: custom pattern-matching.
Most languages with pattern-matching have a fixed set of rules and patterns for how everything works.
Not so with Lux.
Lux provides a set of default mechanisms, but by using macros where patterns are located, when can expand those macro calls to get the myriad benefits they offer.
But enough chit-chat.
Let's see them in action.
Note: The following examples assume you .require the ["^" library/lux/meta/macro/pattern] module.
Pattern-matching macros in the Standard Library
(when (list 1 2 3)
(list x y z)
{.#Some (+ x (* y z))}
_
{.#None})
You may remember how annoying it was to pattern-match against lists in the Chapter 5 example.
Well, when will expand any normal macros inside your patterns before proceeding to pattern-match against the input data.
... Multi-level pattern matching.
... Useful in situations where the result of a branch depends on further refinements on the values being matched.
... For example:
(when (split (size static) uri)
(^.multi {.#Some [chunk uri']}
[(text::= static chunk) #1])
(match_uri endpoint? parts' uri')
_
{.#Left (format "Static part " (%.text static) " doesn't match URI: " uri)})
... Short-cuts can be taken when using boolean tests.
... The example above can be rewritten as...
(when (split (size static) uri)
(^.multi {.#Some [chunk uri']}
(text::= static chunk))
(match_uri endpoint? parts' uri')
_
{.#Left (format "Static part " (%.text static) " doesn't match URI: " uri}))
I love multi.
It's one of those features you don't need often, but when you do, it saves the day.
The possibilities are endless when it comes to the refinement you can do, and when you consider what you'd have to do to get the same results without it, it's easy to see how much code it saves you.
... Allows you to simultaneously bind and de-structure a value.
(def (hash (^.let set [element_hash _]))
(list#mix (function (_ elem acc)
(n.+ (at element_hash hash elem) acc))
0
(set.list set)))
let is for when you want to deal with a value both as a whole and in parts.
... Same as the "use" macro, but meant to be used as a pattern-matching macro for generating local bindings.
... Can optionally take an aliasing text for the generated local bindings.
(def .public (range (.open "[0]") from to)
(All (_ a) (-> (Enum a) a a (List a)))
(range' <= succ from to))
.open allows you to open structures using local variables during pattern-matching.
It's excellent when taking structures as function arguments, or when opening structures locally in let expressions.
... Or-patterns.
(type: .public Day
(Variant
{#Sunday}
{#Monday}
{#Tuesday}
{#Wednesday}
{#Thursday}
{#Friday}
{#Saturday}))
(def (weekend? day)
(-> Day Bit)
(when day
(^.or {#Saturday} {#Sunday})
true
_
false))
or patterns allow you to have multiple patterns with the same branch.
It's a real time-saver.
... It's similar to do-template, but meant to be used during pattern-matching.
(def (beta_reduce env type)
(-> (List Type) Type Type)
(when type
{.#Primitive name params}
{.#Primitive name (list#map (beta_reduce env) params)}
(^.with_template [<tag>]
[{<tag> left right}
{<tag> (beta_reduce env left) (beta_reduce env right)}])
([.#Sum]
[.#Product]
[.#Function]
[.#Apply])
(^.with_template [<tag>]
[{<tag> old_env def}
(when old_env
{.#End}
{<tag> env def}
_
type)])
([.#UnivQ]
[.#ExQ])
{.#Parameter idx}
(maybe.else type (list.item idx env))
{.#Named name type}
(beta_reduce env type)
_
type))
with_template is or's big brother.
Whereas or demands that you provide the exact patterns you want (and with a single branch body), with_template lets you provide templates for both the patterns and bodies, and then fills the blanks with all the given parameters.
You can save yourself quite a lot of typing (and debugging) by reusing a lot of pattern-matching code with this macro.
It's a great asset!
... Allows destructuring of streams in pattern-matching expressions.
... Caveat emptor: Only use it for destructuring, and not for testing values within the streams.
(let [(stream.pattern x y z _tail) (some_stream_function 1 2 3)]
(func x y z))
stream.pattern hails from the library/lux/data/collection/stream module, and it's quite special, because it allows you to de-structure something you normally wouldn't be able to: functions.
You see, Lux streams (as defined in library/lux/data/collection/stream) are implemented using functions.
The reason is that they are infinite in scope, and having an infinite data-structure would be... well... impossible (unless you manage to steal a computer with infinite RAM from space aliens).
Well, no biggie!
stream.pattern does some black magic to make sure you can de-structure your stream just like any other data-structure.
How to make your own
The technique is very simple.
Just create a normal macro that will take as a first argument a form containing all the parameters to your pattern-matching macro.
Its second argument will be the body associated with the pattern.
After that, you'll receive all the branches (pattern + body) following the call to PM-macro.
You can process all your inputs as you wish, but in the end you must produce an even number of outputs (even, because the outputs must take the form of pattern+body pairs).
These will be further macro-expanded until all macros have been dealt with and only primitive patterns remain, so when can just go ahead and do normal pattern-matching.
You may wonder: why do I receive the body?
The answer is simple: depending on your macro, you may be trying to provide some advance functionality that requires altering (or replicating) the code of the body.
For example, ^.or copies the body for every alternative pattern you give it, and ^.with_template must both copy and customize the bodies (and patterns) according to the parameters you give it.
But receiving the next set of branches takes the cake for the weirdest input.
What gives?
It's simple: some macros are so advanced that they require altering not just their bodies, but anything that comes later.
A great example of that is the ^.multi macro.
^.multi performs some large-scale transformations on your code which require getting access to the rest of the code after a given usage of ^.multi.
However, most of the time, you'll just return the branches (and sometimes the body) unchanged.
To make things easier to understand, here is the implementation of the or macro, from the library/lux/meta/macro/pattern module:
(def .public or
(pattern
(macro (_ tokens)
(when tokens
(list.partial [_ {.#Form patterns}] body branches)
(when patterns
{.#End}
(///.failure (..wrong_syntax_error (symbol ..or)))
_
(.let [pairs (.|> patterns
(list#each (function (_ pattern) (list pattern body)))
list#conjoint)]
(///#in (list#composite pairs branches))))
_
(///.failure (..wrong_syntax_error (symbol ..or)))))))
Appendix D: The art of piping
I'm a big fan of piping.
No kidding.
Piping is my favorite way of writing code.
It's almost like a game to me.
I try to figure out ways to get my code to be more pipe-sensitive, to see how far I can get while piping my code.
My personal record is 14 steps.
Note: The following examples assume that you .require the ["|" library/lux/control/pipe] module.
Piping macros in the standard library
Anyhow, after looking at some of the innovations of Clojure in the piping department, I decided to come up with my own tricks to try to get Lux to become a piping superpower.
I added the library/lux/control/pipe module, which contains several macros meant to be used within the |> macro, and which extend it with awesome capabilities.
Take a look at these babies:
... Loops for pipes.
... Both the testing and calculating steps are pipes and must be given inside tuples.
(|> 1
(|.loop [(n.< 10)]
[++]))
loop takes a test tuple and a body tuple.
The reason is that each of those tuples represents the steps on an implicit piping macro (oh, yeah!).
So [(n.< 10)] is like (|> value (n.< 10)), and [++] is like (|> value ++).
Which value? Whatever has been piped into loop from the underlying |> (in this case, the value 1).
... Branching for pipes.
... Both the tests and the bodies are piped-code, and must be given inside a tuple.
... If a last else-pipe isn't given, the piped-argument will be used instead.
(|> 5
(|.cond [i.even?] [(i.* 2)]
[i.odd?] [(i.* 3)]
... else branch
[(|.new -1 [])]))
We have looping, and now we have branching; with a cond-inspired piping macro (complete with else branch, just in case).
But what's that thing over there? That new thing?
Well, it's another piping macro. Of course!
... Ignores the piped argument, and begins a new pipe.
(|> 20
(i.* 3)
(i.+ 4)
(|.new 0 [++]))
new establishes a new piping sequence that ignores any previous one.
Useful in certain kinds of situations.
... Gives a name to the piped-argument, within the given expression.
(|> 5
(|.let @ (+ @ @)))
let binds the current value piped into it so you can refer to it multiple times within its body.
Pretty nifty, huh?
... Pattern-matching for pipes.
... The bodies of each branch are NOT pipes; just regular values.
(|> 5
(|.when 0 "zero"
1 "one"
2 "two"
3 "three"
4 "four"
5 "five"
6 "six"
7 "seven"
8 "eight"
9 "nine"
_ "???"))
Yeah, that's right!
I just couldn't resist rolling full-blown pattern-matching into this.
You'll thank me later.
... Monadic pipes.
... Each step in the monadic computation is a pipe and must be given inside a tuple.
(|> 5
(|.do identity.monad
[(i.* 3)]
[(i.+ 4)]
[+]))
And just to show you I'm serious, I did the unthinkable.
Piped monadic expressions!
How to make your own piping macros
They're easier to make than pattern-matching macros.
All you need is a macro that takes anything you want as parameters, but always takes as its last argument the computation so far, as it has been constructed by the |> macro prior to the call to your piping macro.
As an example, here's the definition for let>:
(def .public let
(syntax (_ [binding <code>.any
body <code>.any
prev <code>.any])
(in (list (` (.let [(, binding) (, prev)]
(, body)))))))
All this looks like madness, but I just couldn't contain myself.
Piping is one of the few ways of writing code that just amuses me whenever I do it.
These macros can keep you in the flow while you're writing complex code, so you don't have to switch so much between piping-code and non-piping-code.
Oh... and did I mention the |>> macro?
It generates for you a single-argument function that will immediately pipe its argument through all the steps you give it.
(only (|>> (member? forbidden_definitions)
not)
all_definitions)
Appendix E: Lux implementation details
If you read Chapter 6, you encountered Lux's funny way of encoding variants, tuples and functions.
You may be wondering: how can this possibly have good performance?
And: what benefit can this possible have?
I'll tackle those questions one at a time.
How can this possibly have good performance?
First, let me explain how things get compiled down in the JVM.
Tuples are compiled as object arrays. That means an n-tuple is (roughly) an n-array.
The reason why I say _"roughly"_ will be explained shortly.
Variants, on the other hand, are 3-arrays.
- The first element is the int value of its associated tag.
- The second element is a kind of boolean flag used internally by the Lux run-time infrastructure.
- The third element contains the variant's value.
Finally, functions produce custom classes, and function values are just objects of those classes.
These classes contain everything the function needs:
- its compiled code.
- its environment/closure.
- any partially-applied arguments it may have.
How, then, can all of this be made efficient?
Does applying a function f to arguments a, b and c create intermediate function values because you can only apply it one argument at a time?
Do tuples consume a lot of memory because everything gets nested?
Not really.
With regards to tuples, remember what I said: an n-tuple is (roughly) an n-array.
If you write [#0 12 -34 +56.78 "nine"], Lux will actually compile it down as a 5-array, instead of a series of nested 2-arrays.
However, if you have a variable foo which contains the last two arguments, and you build your tuple like [#0 12 -34 foo], Lux will compile it as a 4-array, with the last element pointing to the [+56.78 "nine"] sub-tuple.
But, as I said in Chapter 6, Lux treats both the same.
How does that work?
Well, Lux knows how to work with both flat and nested tuples and it can do so efficiently; so ultimately it doesn't matter.
It will all be transparent to you.
When it comes to variants, the situation is similar in some ways, but different in others.
Regardless, Lux also knows how to work with the different cases efficiently (which is important for pattern-matching, not just for variant/tuple construction).
Finally, we have to consider functions.
Merging nested functions into a single one that can work like all the nested versions turns out to be pretty easy.
Just allocate enough space for all the (potentially) partially-applied arguments, plus space for the environment/closure.
If you invoke the function with all the arguments, you just run it.
If you invoke it with less than needed, you just use the space you have to store the partial arguments and generate a single new instance with the extra data (instead of generating a new function object for every argument you apply).
And if you're invoking a partially applied function, then you run it with the partial arguments and the new arguments.
Piece of cake.
What benefit can this possible have?
I already explained in Chapter 6 how the nested nature of Lux functions enables partial application (a useful day-to-day feature that saves you from writing a lot of boilerplate).
What about variants and tuples?
Well, the cool thing is that this makes your data-structures composable, a property that enables you to implement many really cool features.
One that I really like and has turned out to be very useful to me, is that you can use combinators for various data-types that produce single bits of data, and you can fuse them to generate composite data-types, with minimal plumbing.
You can see _combinators_ as functions that allow you to provide an extra layer of functionality on top of other components, or that allow you to fuse components to get more complex ones.
Here are some examples from the library/lux/ffi module, where I have some types and code-parsers for the many macros implemented there:
(type: .public Privacy
(Variant
{#PublicP}
{#PrivateP}
{#ProtectedP}
{#DefaultP}))
(def privacy_modifier^
(Parser Privacy)
(let [(open "[0]") <>.monad]
($_ <>.or
(<code>.this! (' "public"))
(<code>.this! (' "private"))
(<code>.this! (' "protected"))
(in []))))
Here, I have a variant type, and I'm creating a code-parser that produces instances of it by simply combining smaller parsers (that just produce unit values, if they succeed) through the <>.or combinator.
These code-parsers and combinators are defined in the `library/lux/control/parser/code` module, and the `library/lux/control/parser` module.
<>.or is a combinator for generating variant types.
Its tuple counterpart is called <>.and (also, remember that records are tuples, so you'd use the same function).
This wouldn't be possible if variant types weren't nested/composable; forcing me to write custom ad-hoc code instead of taking advantage of common, reusable infrastructure.
Here's an example of <>.and in action:
... From library/lux/target/jvm/type
(type: .public Argument
[Text (Type Value)])
... From library/lux/ffi
(def (argument^ type_vars)
(-> (List (Type Var)) (Parser Argument))
(<code>.tuple (<>.and <code>.local_symbol
(..type^ type_vars))))
The cool thing is that these combinators show up not just in syntax parsers, but also in command-line argument parsing, lexing, concurrenct/asynchronous operations, error-handling and in many other contexts.
The nested/composable semantics of Lux entities provide a flexibility that enables powerful features (such as this) to be built on top.
Appendix F: Implicit polymorphism
If you've used Lux's interfaces and implementations already (with the at macro), you've probably noticed that you need to pass around the specific implementations you need every time you want to call some interface's method.
That can become tiresome if you need to do it all the time, and specially if you come from languages that do method-selection for you automatically.
Object-oriented languages do polymorphism in an easy way, because they link objects to the method table of their associated classes, and when you call a method on an object, the run-time system can figure out where the code that needs to be run lies within the program's memory.
Languages with type-classes, such as Haskell, perform that look-up at compile-time, by using the type-information present in the compilation context to figure out which implementation (or instance) of a type-class is suitable to each particular circumstance.
Lux, on the other hand, forces you to be specific about the implementations that you're going to use.
While that gives you a level of power and flexibility you wouldn't otherwise have in other languages, it also introduces the problem that when what you want doesn't warrant that level of power, you have to pay the tax it involves nonetheless.
But, that sounds like a raw deal.
Why do you have to pay for something you're not taking advantage of?
Clearly, there is an asymmetry here.
It is a feature that is most useful in the few instances when you want full power.
At any other point, it's a hindrance.
Well... there is an alternative.
The Lux Standard Library includes a module called library/lux/type/implicit, which provides a macro called a/an (or, alternatively, a or an), that serves as an easier-to-use alternative to the at macro.
What it does is that instead of requiring the implementation you want to use, it only requires the name of the method you want to call and the arguments.
Then, at compile-time, it does some type-checking and some look-ups and selects an implementation for you that will satisfy those requirements.
That implementation can come from the local-var environment, from the definitions in your own module, or even from the exported definitions of the modules you're importing.
That way, you can use at whenever you need precision and power, and use a/an whenever you're doing more lightweight programming.
Fantastic!
This is how you'd use it:
... Equality for nats
(at nat.equivalence = x y)
... vs
(an = x y)
... Equality for lists of nats
(at (list.equivalence nat.equivalence) =
(list.indices 10)
(list.indices 10))
... vs
(an = (list.indices 10) (list.indices 10))
... Functor mapping
(at list.functor each ++ (list.indices 10))
... vs
(an each ++ (list.indices 10))
Thanks to implicit polymorphism, you don't have to choose between power and ease of use.
Just do a static-import of the library/lux/type/implicit module, and you'll get the a/an available and ready for action.
Appendix G: Regular expressions
Working with text is a pretty common and fundamental thing in day-to-day programming.
Lux's approach to doing it is with the use of composable, monadic text parsers.
The idea is that a parser is a function that takes some text input, performs some calculations which consume that input, and then returns some value, and (the remaining) unconsumed input.
Of course, the parser may fail, in which case the user should receive some meaningful error message to figure out what happened.
The library/lux/control/parser/text library provides a type, and a host of combinators, for building and working with text parsers.
(type .public Offset
Nat)
(type .public Parser
(//.Parser [Offset Text]))
... And from library/lux/control/parser
(type .public (Parser s a)
(-> s (Try [s a])))
A good example of text parsers being used is the library/lux/data/format/json module, which implements full JSON serialization.
However, programmers coming from other programming languages may be familiar with a different approach to text processing that has been very popular for a number of years now: regular expressions.
Regular expressions offer a short syntax to building text parsers that is great for writing quick text-processing tools.
Lux also offers support for this style in its library/lux/data/text/regex module, which offers the regex macro.
The regex macro, in turn, compiles the given syntax into a text parser, which means you can combine both approaches, for maximum flexibility.
Here are some examples of regular expressions:
... Literals
(regex "a")
... Wildcards
(regex ".")
... Escaping
(regex "\.")
... Character classes
(regex "\d")
(regex "\p{Lower}")
(regex "[abc]")
(regex "[a-z]")
(regex "[a-zA-Z]")
(regex "[a-z&&[def]]")
... Negation
(regex "[^abc]")
(regex "[^a-z]")
(regex "[^a-zA-Z]")
(regex "[a-z&&[^bc]]")
(regex "[a-z&&[^m-p]]")
... Combinations
(regex "aa")
(regex "a?")
(regex "a*")
(regex "a+")
... Specific amounts
(regex "a{2}")
... At least
(regex "a{1,}")
... At most
(regex "a{,1}")
... Between
(regex "a{1,2}")
... Groups
(regex "a(.)c")
(regex "a(b+)c")
(regex "(\d{3})-(\d{3})-(\d{4})")
(regex "(\d{3})-(?:\d{3})-(\d{4})")
(regex "(?<code>\d{3})-\k<code>-(\d{4})")
(regex "(?<code>\d{3})-\k<code>-(\d{4})-\\0")
(regex "(\d{3})-((\d{3})-(\d{4}))")
... Alternation
(regex "a|b")
(regex "a(.)(.)|b(.)(.)")
Another awesome feature of the regex macro is that it will build fully type-safe code for you.
This is important because the groups and alternations that you use in your regular expression will affect the type of the regex expression.
For example:
... This returns a single piece of text
(regex "a{1,}")
... But this one returns a pair of texts
... The first is the whole match: aXc
... And the second is the thing that got matched: the X itself
(regex "a(.)c")
... That means, these are the types of these regular-expressions:
(is (Parser Text)
(regex "a{1,}"))
(is (Parser [Text Text])
(regex "a(.)c"))
The benefits of parsers are that they are a bit easier to understand when reading them (due to their verbosity), and that they are very easy to combine (thanks to their monadic nature, and the combinator library).
The benefits of regular expressions are their familiarity to a lot of programmers, and how quick they are to write.
Ultimately, it makes the most sense to provide both mechanisms to Lux programmers, and let everyone choose whatever they find most useful.
Appendix H: Aedifex
Aedifex is a fairly basic build tool, but it offers the necessary commands to handle a normal Lux project.
It offers a small set of commands, and a few convenience features.
lux version
=>
0.07.00
This command tells you the version of Aedifex you're using.
This version is also the same as the version of the Lux compiler Aedifex has been designed to work with.
So, you can also think of this as being the version of the language that is best supported by Aedifex (although not necessarily the only version that is supported).
lux clean
=>
Successfully cleaned target directory: target
This command cleans up any compilation artifacts and caching artifacts left over by previous compilations/builds of your Lux project.
lux pom
=>
Successfully created POM file: pom.xml
This command generates a Maven POM file that reflects the contents of the project.lux file made for Aedifex.
lux deps
=>
[?] Fetching com.github.luxlang:lux-jvm-0.6.5 from "~/.m2/repository"
[O] Found com.github.luxlang:lux-jvm-0.6.5 at "~/.m2/repository"
Local successes: 0: "com.github.luxlang:lux-jvm-0.7.0"
Local failures:
Remote successes:
Remote failures:
This commands fetches all dependencies from the available repositories (plus the local Maven repository/cache), and installs any dependencies that had to be fetched remotely into the local Maven repository/cache.
lux install
=>
Successfully installed the project locally.
This command packages your project into a TAR archive and installs it into the local Maven repository/cache, to be used as a dependency.
lux deploy <deploy_repository> <user_name> <password>
## For example:
lux deploy snapshots foo bar_baz_quux
=>
Successfully deployed the project.
This command packages your project into a TAR archive and deploys it to a remote Maven repository, to be used as a dependency.
lux build
This command build your project into an executable program, named program.(jar|js|lua|py|rb) depending on the chosen compilation target, and located under the target directory (target by default).
lux test
This command build your project into an executable program, and then executes it and display any STDOUT/STDERR logging output on the terminal/console.
This is the main mechanism to run automated tests for your Lux project.
lux auto build
## OR
lux auto test
This works the same as the normal versions of the command, but Aedifex also watches over the files in your source directories and every time a file changes, it automatically builds/tests the project for you.
This is extremely useful when fixing typing errors iteratively, or when debugging errors raised during your tests, as it saves you having to constantly re-build/re-run the project every time you want to try the latest changes.
lux with <profile> <command>
## For example:
lux with jvm with bibliotheca auto test
This command composes non-default profiles with the default one to generate the profile to use when executing a command.
This allows you to segregate useful configuration into different profiles and then combine them based on what you need at a given time.
Now that we have seen the available commands, it would be useful to see an annotated example project.lux file to see what bits of configuration it can contain.
["" ... The empty text ("") is used to specify the default profile.
[... An optional identity for the project.
... It can also be specified or overriden in a non-default profile.
... This will be the name given to the project when installed/deployed as a dependency.
"identity" ["com.github.luxlang" "stdlib" "0.6.5"]
... Every piece of information, and the whole "info" bundle, are optional.
"info" ["url" "https://github.com/LuxLang/lux"
"scm" "https://github.com/LuxLang/lux.git"
"licenses" [["name" "Lux License v0.1.1"
"url" "https://github.com/LuxLang/lux/blob/master/license.txt"
"type" "repo"]]
... "organization" [["name" "Lux Foundation"
... "url" "http://example.com/lux_foundation"]]
"developers" [["name" "Eduardo Julian"
"url" "https://github.com/eduardoejp"
... "organization" ["name" "Lux Foundation"
... "url" "http://example.com/lux_foundation"]
]]
... "contributors" [["name" "Eduardo Julian"
... "url" "https://github.com/eduardoejp"
... "organization" ["name" "Lux Foundation"
... "url" "http://example.com/lux_foundation"]]]
]
... An optional list of repositories you can deploy to, given aliases so they're easy to refer to with the "deploy" command.
"deploy_repositories" ["snapshots" "https://oss.sonatype.org/content/repositories/snapshots/"
"releases" "https://oss.sonatype.org/service/local/staging/deploy/maven2/"]
... An optional list of repositories to use for fetching remote dependencies.
... Additionally, there is an implicit repository being used, which is https://repo1.maven.org/maven2/
... So, even if the "repositories" list were to be empty, you'd still have access to the default repository.
"repositories" ["https://oss.sonatype.org/content/repositories/snapshots/"
"https://oss.sonatype.org/service/local/staging/deploy/maven2/"]
... The different directories to look for source code. The default is described below.
... "sources" ["source"]
... The directory for storing the build artifacts. The default is described below.
... "target" "target"
]
... The following are alternative profiles to use in various situations.
"jvm"
[... "lux" specifies the dependency to fetch and use as the compiler.
"lux" ["com.github.luxlang" "lux-jvm" "0.6.5" "jar"]
... "dependencies" is an optional list of dependencies to fetch.
... The dependencies have the same shape as when specifying the compiler.
... When omitting the packaging format of the dependency, "tar" will be assumed.
... "dependencies" [["org.ow2.asm" "asm-all" "5.0.3" "jar"]
... ["com.github.luxlang" "stdlib" "0.6.5"]]
... The OS command to use when running JVM tests. The default is described below.
... "java" ["java" "-jar"]
]
"js"
["lux" ["com.github.luxlang" "lux-js" "0.6.5" "js"]
... The OS command to use when running JS tests. The default is described below.
... "js" ["node" "--stack_size=8192"]
]
"python"
["lux" ["com.github.luxlang" "lux-python" "0.6.5" "jar"]
... The OS command to use when running Python tests. The default is described below.
... "python" ["python3"]
]
"lua"
["lux" ["com.github.luxlang" "lux-lua" "0.6.5" "jar"]
... The OS command to use when running Lua tests. The default is described below.
... "lua" ["lua"]
]
"ruby"
["lux" ["com.github.luxlang" "lux-ruby" "0.6.5" "jar"]
... The OS command to use when running Ruby tests. The default is described below.
... "ruby" ["ruby"]
]
"bibliotheca"
["info" ["description" "Standard Library for the Lux programming language."]
"test" test/lux._]
"scriptum"
["info" ["description" "Documentation for the Standard Library for the Lux programming language."]
"program" documentation/lux._
"test" documentation/lux._]
"aedifex"
["info" ["description" "A build system/tool made exclusively for Lux."]
... Parent profiles to this one.
... Specifying them here is like automatically using Aedifex's "with" command.
"parents" ["jvm"]
... The name of the main module of the program.
"program" program/aedifex._
... The name of the main module where the tests are located.
"test" test/aedifex._]]