University of Warwick
Dissertation Report
A Denitive System For The Browser
Author:
Timothy
Monks
Supervisor:
0622387
Dr. Steve
Russ
Keywords: JavaScript, Denitive Notations, Modelling, Web Browser, HTML5,
Compiler, Dependency, EDEN
September 15, 2011
Abstract
Interactive modelling through denitions has been a successful way of Thinking with
the computer in the Empirical Modelling Research group and with a wider community through spreadsheet packages. This project is an attempt to bring generalised
modelling to the browser using the browser-side scripting technology, JavaScript.
The report gives a technical description of the maintainer for state with denitions
and a translator for a language (EDEN), which allows for ecient maintenance of
such a system.
1
Contents
1 Introduction
6
1.1
Report structure
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6
1.2
Acknowledgements
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
1.3
Motivation for Denitive Systems . . . . . . . . . . . . . . . . . . . .
7
1.3.1
Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
1.3.2
Denitive Notations
. . . . . . . . . . . . . . . . . . . . . . .
8
. . . . . . . . . . . . . . . . . . . . . . . . . . .
9
. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
1.4
What is JavaScript?
1.5
Why JavaScript?
2 Related Work
12
2.1
TkEden
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2
The EDEN notation
2.3
WebEden
2.4
. . . . . . . . . . . . . . . . . . . . . . . . . . .
12
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
13
JavaScript Data Binding APIs . . . . . . . . . . . . . . . . . . . . . .
14
3 Aims and Initial Design
3.1
3.2
12
16
Design Thoughts for a Maintainer for State with Denitions
. . . . .
16
. . . . . . . . . . . . . .
16
. . . . . . . . . . . . . . . .
19
. . . . . . . . . . . . . . .
20
3.2.1
Preliminary Work: A JSON DSL for Denitions . . . . . . . .
20
3.2.2
Preliminary Work: Denition Maintainer . . . . . . . . . . . .
21
3.2.3
Change of Approach:
3.1.1
Support for Interactive Development
3.1.2
Associating Denitions and State
Design Thoughts for a Denitive Notation
Representation
Using JavaScript as an Intermediate
. . . . . . . . . . . . . . . . . . . . . . . . . .
21
3.3
Design Thoughts for an End User Interface - JsEden
. . . . . . . . .
23
3.4
Overview of Underlying Structure . . . . . . . . . . . . . . . . . . . .
23
2
4 Description of maintainer.js
25
4.1
Observables
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2
Symbol Table
4.3
Dealing with Observables that have yet to be dened
. . . . . . . . .
26
4.4
Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
4.5
Denition Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . .
30
4.6
Simple JavaScript Agents
. . . . . . . . . . . . . . . . . . . . . . . .
30
4.7
Querying State
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
33
4.8
Mutation of observables
. . . . . . . . . . . . . . . . . . . . . . . . .
34
4.9
Note about Determining Dependencies Dynamically . . . . . . . . . .
34
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 EDEN Translation Scheme
25
26
35
5.1
Number, Character and String Data Types . . . . . . . . . . . . . . .
35
5.2
Observables
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
5.3
Arithmetic, String and Boolean Logic Expressions . . . . . . . . . . .
36
5.4
Dierences Between EDEN and JavaScript's Built In Operators
. . .
36
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
37
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
5.4.1
5.5
Lists
Pointers
5.5.1
List Indexing
. . . . . . . . . . . . . . . . . . . . . . . . . . .
38
5.5.2
List modication statements . . . . . . . . . . . . . . . . . . .
39
5.6
Denitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
5.7
Extracting Denitions in EDEN Form for Formula Inspection
. . . .
39
5.8
Functions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
5.9
Function Arguments
. . . . . . . . . . . . . . . . . . . . . . . . . . .
40
5.10 Function Argument Aliases . . . . . . . . . . . . . . . . . . . . . . . .
42
5.11 Function Local Variables . . . . . . . . . . . . . . . . . . . . . . . . .
43
5.12 Triggered Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
5.13 Control Flow
44
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.14 Verbatim JavaScript Sections
. . . . . . . . . . . . . . . . . . . . . .
45
5.15 Methodology for Developing the Translator . . . . . . . . . . . . . . .
45
3
6 Example Applications
47
6.1
JsEden Prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
6.2
General Approach to Developing Visualisations with JsEden Models .
47
6.3
Example JsEden Model With A Simple Box Visualisation . . . . . . .
49
6.4
Embedding Formulas Into an HTML Document
50
. . . . . . . . . . . .
7 Project Management
52
7.1
Timeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
7.2
Methodology
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
52
7.3
Other commitments . . . . . . . . . . . . . . . . . . . . . . . . . . . .
53
7.4
Version Control and Backups
. . . . . . . . . . . . . . . . . . . . . .
53
7.5
Tools
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
7.6
Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
54
7.7
Legal, social, ethical and professional issues . . . . . . . . . . . . . . .
55
8 Conclusions
56
8.1
Success of targetting EDEN
. . . . . . . . . . . . . . . . . . . . . . .
56
8.2
Potential outside of EDEN . . . . . . . . . . . . . . . . . . . . . . . .
57
9 Further Work
58
9.1
More Model Development
. . . . . . . . . . . . . . . . . . . . . . . .
9.2
Implement Cyclic Dependency Checking
. . . . . . . . . . . . . . . .
58
9.3
Possibilities outside the browser . . . . . . . . . . . . . . . . . . . . .
58
9.4
Collaborative modelling and interaction with already existing tools
.
59
9.5
Modelling with hierarchical structures . . . . . . . . . . . . . . . . . .
59
9.6
Support for other notations
. . . . . . . . . . . . . . . . . . . . . . .
60
9.7
Improved JsEden UI
. . . . . . . . . . . . . . . . . . . . . . . . . . .
60
9.8
Improved Error Reporting for EDEN Code . . . . . . . . . . . . . . .
61
9.9
Persistency for the developed models in JsEden
61
4
. . . . . . . . . . . .
58
10 Appendices
64
10.1 Build Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
10.2 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
10.3 Glossary of Technical Terms . . . . . . . . . . . . . . . . . . . . . . .
65
List of Figures
2.1
Screenshot of WebEden . . . . . . . . . . . . . . . . . . . . . . . . . .
14
3.1
Activity for a JavaScript VM over time . . . . . . . . . . . . . . . . .
17
3.2
Activity for a TkEden VM over time
18
3.3
An overview of how EDEN code is used with the JavaScript maintainer 24
4.1
Illustration of objects sharing a common operation via a prototype
object
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
4.2
Example of the Structure used to represent an observable . . . . . . .
27
4.3
Illustration of the symbol table used in the maintainer
. . . . . . . .
28
4.4
Illustration of notication of change to a dependent symbol . . . . . .
31
4.5
An example of agent observation
32
4.6
Illustration of scheduling a triggered action through
5.1
Screenshot of the grammar testing and development environment
6.1
Prototype JsEden Interface
6.2
A simple visualisation using a square
. . . . . . . . . . . . . . . . . .
50
6.3
Web page with embedded formulas that reect the maintainer.js state
51
. . . . . . . . . . . . . . . . . . . .
setTimeout
. .
33
. .
46
. . . . . . . . . . . . . . . . . . . . . . .
47
5
1
Introduction
1.1 Report structure
The project undertaken was to implement an interpreter for denitive notations that
would run in a modern browser.
This report is meant to provide a grounding so
that the reader may understand the project's topic and motivation, as well as documentation and analysis of the project itself. The report covers some rather technical
aspects of the implementation as well as possible future uses for the project and also
some higher level discussion on the topics of denitive notations and modelling. The
structure for the report is:
•
Introduction:
The rst section is intended to outline the problem and
provide a proper context to the rest of the text of the report. It will introduce
denitive notations as a tool for model building and software development as
well as the JavaScript platform.
•
Related work:
This section goes into some deeper analysis of already exist-
ing denitive systems and related libraries and frameworks that already exist
for JavaScript.
Particular attention is paid to how the current JavaScript
approaches dier from the interactive modelling approach usually associated
with denitive scripts.
•
Project Aims and Initial Design:
This section outlines the project aims
as well as the design process for the system to be built.
•
Description of maintainer.js:
This section is an in detail description of
the mechanics for the maintainer for state through denitions, including the
API for the maintainer, the underlying structures used and the state caching
system.
•
Project Management:
This section outlines the time line for the project,
as well as the tools used and any issues encountered.
•
Conclusions:
This section is a retrospective on the work done, including an
analysis of the success of the project.
•
Further Work:
This section covers potential extensions to the work done in
this project.
6
1.2 Acknowledgements
The work done in this project is my own. However, there are many people I would
like to thank for helping me get through this project, which has been technically
and more so personally dicult for me at times. While there are too many people
to thank individually, here is a summary:
•
The project supervisor Steve Russ and also Meurig Beynon, both of whom
oered much inspiration through their interest in my work, as well as great
support in something I personally nd dicult - writing up.
•
My girlfriend, Jess, who is far better at reminding me to nish things than my
own quiet conscience.
•
My family, for support and perspectives from their own experiences with similar problems in the past and for allowing me to have somewhere to live.
•
The sta at the department of computer science for giving me the chance to
complete this project and my degree.
•
Far too many students from the department of computer science and the computing society for reminding me that people are interested in what I do and
do interesting things with computers, and for having more faith in me than I
do!
1.3 Motivation for Denitive Systems
This section provides a small discussion on models leading to the motivations for
using computers and denitive systems for model construction.
1.3.1 Models
A model is a construction that helps humans reason about the world. For example,
understanding the combination of quantities can be helped via the combination
of symbols and sets of rules (logic, or mathematics), or using a physical device
like an abacus. In either case there is some external experience from the referent
phenomenon (the thing we are interested in understanding) that we can somehow
use to make conclusions about the referent.
7
As well as helping us reason about things, models also help us communicate about
the world. This can depend on the representation used for the modelling, some are
more convenient for communication than others.
History has seen the development of increasingly complex models (mostly mathematical systems, as opposed to increasingly complex abacuses), which have allowed
humans to reason about dicult things like the spread of infectious disease and next
week's weather. The development of computers has allowed us to solve the complex
equations involved in these models in very short periods of time, thus, computers
are an important part of understanding complex systems.
1.3.2 Denitive Notations
A denitive notation is somewhat like a programming language, it is a tool to craft
a source of experience which utilises computer hardware. One might use either tool
to model a familiar real world phenomena (e.g. a simulation of a ball falling into
a uid), or go the other way, and create something that is unfamiliar (e.g. reverse
the gravitational eect on the ball).
The key feature for a denitive notation is the presence of denitive statements,
which take the form:
definitive statement ::= symbol “is” formula
These statements dene a particular observable in terms of a formula which may
contain other observables.
In order to build a model using these sorts of denitions, a maintainer is needed
that ensures that the model can never be observed in a state that is unfaithful to
the current set of denitions. This job is very amenable to formal reasoning, so a
computer program is appropriate for carrying this out.
Modelling through denitions is already quite popular in spreadsheet packages.
While similar to programming languages in that they can have imperative parts
for performing algorithms, they generally are found in tools which give a much more
interactive experience in which the model state is much more tangible and immediately modiable. The experience is much more like using a multimedia tool like
PhotoShop, where your actions have much more observable consequences, as opposed to a program where there is a complex relation between your modication to
the program and the behaviour of the program.
8
1.4 What is JavaScript?
This short section is intended to clear up what JavaScript is and provide enough understanding of the features of JavaScript such that the implementation of a denitive
system can be explained later.
JavaScript is a technology which allows for much more dynamic content in a web
page than would be possible from just rendering HTML. Though the name is very
similar to the programming language Java, the language itself is in fact very distinct
except for the fact that both of them have similar syntax to the C programming
language. Apart from this it has the following distinguishing features:
•
Dynamically typed
•
Duck typing
•
First class functions
•
Object Oriented through use of object prototypes
Generally JavaScript is run via a virtual machine, through direct interpretation and
more recently through clever JIT (Just In Time) compilation techniques. The use of
these techniques has turned JavaScript into a much more powerful tool in modern
browsers than it had been previously, especially when compared to alternative technologies such as Flash, Java or SilverLight which have run on sophisticated VMs for
quite some time.
JavaScript is embeddable directly in web pages simply by using HTML
<script>
tags. It does not require any plugins to be installed by the end user of the webpage,
just that they have a browser which supports JavaScript (and have not explicitly
disabled it), which includes the vast majority of browsers today. To make this a bit
clearer, here is a simple example of embedding some JavaScript that changes the
content of a webpage every second:
<html>
<head>
<script type="text/javascript">
var counter = 0;
window.onload = function() {
setInterval(function() {
counter++;
document.body.innerHTML = counter;
9
}, 1000);
};
</script>
</head>
<body>
</body>
</html>
The key things to notice in the listing are:
•
The script is embedded directly in the web page
•
The use of `function()' allows for writing function values, which in turn
allows for the use of higher order functions such as
setInterval
• setInterval takes a procedure to run and an interval, in milliseconds, which
is how frequently the passed procedure should be called.
•
It is possible to easily manipulate the DOM (Document Object Model) of a
document.body.innerHTML
<body> section of the page
webpage from JavaScript, e.g. assignment to
allows mutation of the content in the
1.5 Why JavaScript?
There are several other technologies that allow for development of rich web applications. The other notable technologies are:
•
Adobe Flash
•
Java Applets
•
Microsoft Silverlight
JavaScript was chosen over these for a few reasons:
•
Superior cross platform support. Flash will not run on iOS devices (iPhone,
iPad) for example, and has poor support under GNU/Linux.
run on iOS devices.
Java will not
Silverlight has some support for platforms other than
Microsoft Windows through a free implementation of the .NET runtime called
Mono, but is not ocially supported.
10
•
No plugins required. JavaScript applications work out of the box with any
modern browser. There is no need to manually download any plugins, which
lowers the level for entry for applications written in JavaScript.
•
Event based programming model.
JavaScript allows you to dene handlers
which execute whenever some event happens. These handlers are eectively
observing agents.
•
JavaScript can interact well with webpages. One interesting aspect that might
be explored is using a lightweight JavaScript maintainer to demonstrate the
ideas of dependency in an informational webpage.
•
Simple toolchain.
All that is needed to modify an application written in
JavaScript is a text editor. Refreshing the browser will allow you to use the
updated application.
This could prove valuable for lowering the barrier of
entry for new developers.
11
2
Related Work
This project is mostly a practical one, so while there is limited relevant literature
(except in maybe the ecient implementation of compilers, which will be relevant
later on), it is important to examine already existing systems which use denitive
notations as these were the context in which the design decisions during the project
were made.
2.1 TkEden
TkEden is a tool developed at the University of Warwick which allows for interactive model construction using denitive notations. The name TkEden reects the
underlying interpreter for denitive notations called EDEN, and the toolkit used for
the user interface, Tk. It has a very large history of use by students in the University, leading to a large archive of models which are free to obtain and use. The core
concepts in TkEden are:
•
Observables - all of the model state is captured through named values called
observables
•
Dependency - Observables can be dened in terms of each other, and then
kept consistent by the maintainer
•
Agency - Agency is what causes change in the model state, and can happen
by human intervention or by triggered actions which act as simple machine
based agents watching for changes in particular observables
2.2 The EDEN notation
The TkEden tool infact supports several notations for use in dierent contexts.
The main notation (which most of the other notations are in fact translated to) is
called EDEN. The notation acts as an imperative interface to the underlying engine,
which allows for manipulation of the various observables, dependencies and agency
in a model.
Each statement is like a single interaction with the model state. Some interactions
rene the dependencies that the model adheres to, while others are a direct manipulation of the model state. Others have no interaction with the model state at all,
but cause some side eects somewhere else, e.g. an arithmetic expression with no
12
assignment to an observable will have no eect to the model or an expression which
calls the writeln function to log something to the TkEden console.
The syntax has a similar style to C, as well as supporting imperative style programming (e.g. assignment, loops and other ow control).
Denitions in EDEN take the form:
observable is formula;
Where a formula is some expression, syntactically similar to an expression you might
see in any C like programming language. There is a reasonably accurate documentation of the grammar available from [Beynon(2006)].
2.3 WebEden
WebEden is a tool which allows for modelling with denitions in the browser.
It
was created by Richard Myers as a 3rd year project at the University of Warwick.
The architecture combines a Flash frontend connected up to a modied instance of
TkEden. The important parts of the architecture are:
•
The Flash frontend is all about managing input and presenting a nice interface,
and pushing interactions to the backend via an XML protocol.
•
The interpretation of EDEN code (and code in other notations) and the dependency maintenance all happens on the modied instance of TkEden, which
then communicates any changes back to the UI.
While WebEden was an excellent 3rd year project, and does allow for modelling
through denitions through a remote interface to TkEden, there are some benets
in moving away from the architecture that it uses:
•
Analogue interactions are laggy - Since WebEden relies on transmitting user
interactions over a wire to be processed, and then receiving a response to that,
change triggered by mouse motion seems a lot less continuous than it usually
would.
•
Expensive to scale - WebEden requires a single instance of TkEden for each
user using WebEden. If the maintainer and interpreter existed locally, scaling
would be a lot simpler.
13
Figure 2.1: Screenshot of WebEden
http://www2.warwick.ac.uk/fac/sci/dcs
/research/em/software/webeden/)
(taken from
•
Architecture overly complex - WebEden had sophisticated load balancing and
authentication systems in place. These are denitely important in WebEden,
unfortunately at the University of Warwick these seemed to be problematic,
leading to users having problems in connecting to instances of TkEden hosted
in the department of computer science. This could possibly be xed with the
proper knowledge of WebEden and some negotation with the department, but
it could be avoided entirely if we removed the reliance on a remote dependency
engine an interpreter.
2.4 JavaScript Data Binding APIs
There are a few libraries that already exist for JavaScript which support something
resembling modelling with denitions.
They use a technique called data binding
to tackle a particular concern - updating some visual representation of data from
an underlying model whenever change occurs to the model.
This is analogous to
triggered actions in EDEN, however, these systems generally have no concern for
maintaining state such that a set of denitions is always faithful. Some frameworks
of interest are:
•
backbone.js
14
•
Knockout
•
SproutCore
Data bindings can also be found in other environments such as Adobe FLEX or
when using Key-Value Observation with Objective C.
15
3
Aims and Initial Design
This section provides both an outline of the key aims and objectives as well as a
more granular break down of sub-aims and sub-objectives, so that it is clear what
is trying to be achieved in the following sections.
The overall aim of the project is to attempt to prototype a system for interactive
modelling with denitions, that works in the browser. The main objectives for the
project are:
•
Build a system that maintains state through denitions that runs in modern
web browsers
•
Develop support for a denitive notation for the system
•
Explore possible applications of both
3.1 Design Thoughts for a Maintainer for State with Denitions
3.1.1 Support for Interactive Development
Modelling with denitions is an interactive process where the model is rened
through continuous observation and action.
This is a discussion of how the cho-
sen platform, JavaScript, can allow such a work ow.
A JavaScript engine executes in a single thread, and is asleep most of the time. In
order for computation to happen, some event must trigger it rst. Some examples
of events are a timer nishing, user interaction or receiving an HTTP response. The
timeline in Figure 3.1 illustrates this, where the shaded areas correspond to activity
in the machine.
This is similar to the activity of TkEden's virtual machine. Until agency spontaneously renes the state, the machine can stay asleep. The important thing is that
the TkEden virtual machine ensures that before the state can be observed again,
no observables have associated values which are inconsistent with their denitions.
Figure 3.2 shows how the activity of the TkEden VM can be construed as similar
to the JavaScript one.
Further to this, the TkEden virtual machine needs to notify triggered actions of any
state that was modied by the last spontaneous update. These then execute in some
undened order (but notably not in parallel).
16
activity
time
page
load
user
interaction
user
interaction
Figure 3.1: Activity for a JavaScript VM over time
17
dealing with dependency maintenance and
queueing triggered actions
time
script
loaded
agency
agency
Figure 3.2: Activity for a TkEden VM over time
18
Now, looking at the similarity between the operation of the TkEden VM and the
JavaScript VM, it seems that the JavaScript VM is already a good platform for
interactive development in the style that TkEden supports. The JavaScript process
carries a persistent state with it, and indeed it is very possible to interact with
that state interactively and observe it change. The key dierence is that there is no
notion of a denition attached to a variable, variables only take values. The crux of
the work is to provide some structure for associating denitions with observables, as
before this is done there is really no concept of maintaining the values of observables
based on denitions. It is also desirable to manage the `values' and `denitions' of
observables eciently, i.e. whenever possible, avoid re-computation of the `value' of
an observable based on some denition, though that is more of an implementation
detail than a conceptual problem.
3.1.2 Associating Denitions and State
A simple way to work around the problem of variables not having both a value and
a denition is to make the value stored in a variable model an observable, by using
an aggregate which contains the denition for the observable as well some computed
value for it. JavaScript supports aggregate types as collections of key value pairs,
called Objects. An example of an observable modelled with an object is:
{
value: 2,
definition: function(context) { return 2; }
}
Alternatively it may be preferable to store all the observable values and all the
observable denitions in separate data structures.
There are no hooks in JavaScript for when a variable changes. However, if we always
ensure that updates to state happen through calls to particular functions, we can
ensure that the implementation of these functions guarantees that it is impossible
to observe inconsistent state, as well as notifying any machine agents of changes in
state.
To summarise:
•
JavaScript's execution model already a great match for TkEden's interactive
development
19
•
The JavaScript process carries with it a persistent state which can be interactively modied and observed
•
JavaScript variables have no concept of a denition like TkEden observables,
so some way of modelling this needs to be in place
•
JavaScript doesn't provide hooks for mutation of it's state, however if we
ensure that we use particular functions for all change, we can ensure these
changes come with indivisible changes of dependent observables
3.2 Design Thoughts for a Denitive Notation
3.2.1 Preliminary Work: A JSON DSL for Denitions
Before deciding to carry out this work as a dissertation project, a previous denition
maintainer was developed based around it's own JSON based DSL for specifying
denitions. JSON was used for several reasons:
•
JSON is small in amount of syntax and number of concepts to learn, compared
to something like XML
•
JSON is easy to work with in JavaScript since it maps to JavaScript data
structures directly
•
Allows for the possibility of transmitting denitions between processes, e.g.
for working with denitions in a collaborative setting like dtkeden.
In this system, formulas were represented by a list of JSON Objects, where Objects
represented symbols and operators in the language.
Each Object's meaning was
determined by a particular eld called `type'. An example of a formula would be:
[
{
‘type’: ‘Symbol’,
‘value’: ‘add’
},
{
‘type’: ‘Symbol’,
‘value’: ‘a’
20
}, {
‘type’: ‘Number’,
‘value’: 1
}
]
This structure, when evaluated using a hand written evaluator, would eectively
be equivalent to computing the expression `a + 1' where a is an observable in the
environment that is being maintained.
Further to this a parser for a LISP like dialect was developed, in order to make it
easier to specify formulas for denitions using simple prex notation for functions.
Again this parser was hand written due to the simplicity of the language.
The
previous denition could be written using this language like so:
(add a 1)
The language was developed to the point of supporting lambda (function) expressions and hence the denition of named functions via dening a symbol to be a
lambda expression was possible.
3.2.2 Preliminary Work: Denition Maintainer
As well as having some way of representing denitions, a maintainer also had to be
developed to ensure that observables could be associated with a denition and that
any observables were always faithful to the denitions stored in the system.
The
pattern used for denition re-evaluation is the same as the nal maintainer and in
fact the nal maintainer evolved from the code initially developed at this stage.
3.2.3 Change of Approach: Using JavaScript as an Intermediate Representation
Shortly after starting to develop the previously described system, I discovered CoffeeScript [Ashkenas(2011)] which translates a DSL directly to JavaScript (with the
aim of just providing some syntactic sugar around common JavaScript idioms or
pitfalls), and decided that the approach of translating EDEN to JavaScript was an
interesting idea.
21
The key dierence is that instead of developing a custom data structure that represents a denition (the list structure described in the previous section), the system
would just use native JavaScript functions since they are already the perfect abstraction for a computation, which is essentially what evaluating a denition is.
For example, using a JavaScript function to represent the expression used to demonstrate the JSON powered DSL would result in something like the following:
function() {
return observables.a + 1;
}
Once some JavaScript has been generated, it is possible to dynamically evaluate it
using the builtin
eval
function.
This approach has several advantages over the previously investigated one:
•
No need to design an intermediate representation (IR) capable of representing
enough for EDEN denitions.
Plus, JavaScript is already a great choice of
target language for the EDEN notation, as it is so similar syntactically.
•
No need to develop interpreter for the IR, just use the JavaScript interpreter
(which is also much more ecient in certain cases as it performs Just In Time
compilation [Google(2000)])
As well as this, there would be the benet of the ability to reuse previous models
developed using TkEden, for both learning how direct use in a fully compatible
environment or just as a source of illustrative examples for how to use the EDEN
language.
After this decision, the project had two clear, reasonably independent goals:
1. Develop a maintainer for state with associated functions acting as denitions.
2. Develop a translator for EDEN to JavaScript, relying on methods provided by
the maintainer to interact with some set of observables.
22
3.3 Design Thoughts for an End User Interface - JsEden
Further to the technical work to be done, it would be good to have an interface
that can be used for entering EDEN code interactively and observing the results.
This environment would be called JsEden to reect the targetting of EDEN, and
the underlying implementation using JavaScript. Ideally the interface would look
similar to TkEden, which has quite good facilities for inspecting the current model
state and denitions.
3.4 Overview of Underlying Structure
What follows is an overview of how the individual parts of the JsEden environment
all relate to each other:
The important things to note are:
•
Observables dened using formulas are modelled using some JavaScript data
type
•
Updates to the state happen through function calls, which allows us to intercept them and maintain dependencies eciently, as well as triggering machine
agents Triggered actions
Figure 3.3 shows how the EDEN to JavaScript translator and maintainer.js are
related.
The key things to note are:
•
The EDEN to JavaScript translator is written in JavaScript itself and available
as a function which takes EDEN code as a String and returns JavaScript code
as a String.
•
The JavaScript code is then input into the eval procedure, which runs the code
given to it in the same context as it's callsite. This can cause updates in any
state accessible, most notably it can cause updates via the maintainer.js APIs.
23
other
JavaScript
state
EDEN code
EDEN to
JavaScript
translator
maintainer.js
state
the generated
JavaScript can have
side effects
JavaScript
code
implemented as a JavaScript
function
HTML DOM
state
JavaScript
evaluator
implemented as a builtin function
of JavaScript
Figure 3.3: An overview of how EDEN code is used with the JavaScript maintainer
24
4
Description of maintainer.js
This section gives a more detailed description of the workings of the JavaScript
maintainer, called maintainer.js in this project.
At the start of this project, a small library was developed in order to perform
maintenance of a set of observables, possibly dened in terms of each other, as well
as agent interactions within the world of observables.
This section describes the
underlying workings of the library as a reference for those interested in using it or
creating any sort of derivative work.
This section also goes on to describe other
possible solutions for dependency maintenance and weigh up how much the design
of the chosen solution was inuenced by the desire to support the modelling activity
through the EDEN notation.
The main topics of concern for the library are:
•
How are observables modelled
•
How are observable denitions (formulas) represented
•
How are observables stored and accessed by name
•
How symbols can depend upon, or observe, other symbols
•
How is indivisible dependency implemented eciently
4.1 Observables
Observables should allow for the association of a name with a value as well as a
denition. This was implemented using a JavaScript Object, which is essentially a
key value store as hinted at earlier.
As well as being a key value store, JavaScript objects can come with a prototype
object which is checked in the case where a requested key has no associated value.
This makes it easy to attach common functionality to each Object intended to model
an observable: let them share the same prototype and attach common functionality
to the prototype (see Figure 4.1 for an illustration). In the code developed during
this project, the observables were described as symbols, and they all share the same
Symbol prototype. The following JavaScript code demonstrates how functionality
is tacked on to the Object used as a prototype:
Symbol.prototype.value = function() { ... }
25
The prototype has
common functionality
attached to it
Symbol
Symbol Prototype
Symbol
Symbol
Figure 4.1: Illustration of objects sharing a common operation via a prototype object
Each symbol stores its current value, and its denition. Further to this, it stores
important information such as the symbols that rely on it in their denitions, the
agents that are observing it and also the symbols it relies on and observes.
This
information is used in the cacheing system for observable state. Figure 4.2 illustrates
the structure of the
Symbol type, which just stores this information in various elds.
4.2 Symbol Table
Observables are stored in a symbol table, which maps string identiers to
values.
This allows us to query and modify symbols using a name.
Symbol
This is im-
plemented in a straight forward manner, using an Object where the keys are the
observable names and the values are the Symbols, and is shown in Figure 4.3.
4.3 Dealing with Observables that have yet to be dened
In TkEden it is possible to dene an observable in terms of some other observable
name that currently has had no value or denition associated with it. The API for
26
Symbol
Value
2
Definition
function(c) {
return c.lookup('x') + 1
};
Subscribers
[ ]
Subscribees
[ "x" ]
Observers
[ ]
Figure 4.2: Example of the Structure used to represent an observable
27
Symbol Table
Symbol
Value
Definition
2
function(c) {
return c.lookup('x') + 1
};
y
Subscribers
[ ]
Subscribees
[ "x" ]
Observers
[ ]
x
Figure 4.3: Illustration of the symbol table used in the maintainer
28
the maintainer.js symbol table has been designed to deal with this case. In order to
fetch an observable to query its value or modify it, a call must be made using the
lookup
method. This allows us to check whether the requested symbol has been
observed or dened before: if this is not the case, a new
Symbol
is created for it
and returned. This can then be modied or queried, and will have its initial value
as
undefined.
4.4 Denitions
Denitions are simply represented using a function.
In JavaScript, functions are
values so this makes it simple for dynamic storage of `denitions' by simply storing
a JavaScript function. The convention currently used in the maintainer is that the
rst argument to the function is a symbol table of observables, and the return value
of a function is the `value' of the observable given the current system state. A simple
example of a denition represented this way is:
function(context) {
return context.lookup(’a’).value() + 1;
}
This function, when evaluated, returns 1 greater than the current value of the `a'
observable (assuming the observable `a' has some numeric value currently, else the
result will be an erroneous value).
Any
Symbol
comes with a method to associate a denition with it:
Symbol.prototype.define
Unfortunately functions in JavaScript are not like mathematical functions, they
are not guaranteed side eect free nor are they guaranteed to satisfy referential
transparency (they could rely on some external state for their result), which means
when using the `native' JavaScript interface it is up to the modeller to ensure they
use pure functions for their denitions or they may get some unpredictable (and
unrepeatable) results. This problem can be addressed with an appropriate domain
specic language, however the one implemented later (the EDEN denitive notation)
does not enforce this constraint.
29
4.5 Denition Maintenance
Now that there is a structure which maps all the observables to a function that
can be treated as a `denition' by evaluating it, it is already possible to have a
system where observation of an observable results in a value that is faithful to it's
formula. The most naive solution is to simply evaluate the function used to represent
it's formula on each attempt to inspect it.
However, this is obviously a lot more
wasteful than what can be achieved, also it makes it impossible to implement EDEN
style triggered actions which only trigger when an observable changes, which will be
implemented later.
Therefore, the maintainer attempts to trigger re-evaluation only when necessary.
It does this through a combination of a simple cache mechanism for the results of
formulas, and by allowing the (manual) specication of observables that are relied
upon in a particular formula. These are called dependencies for the formula, and
any change to them should invalidate any cached value for the result of that formula.
An example of using the raw JavaScript api is:
context.lookup(‘a’).define(function(context) {
return context.lookup(‘b’) + 1;
}).subscribe([‘b’]);
The call to
is changed.
subscribe
is what sets up the notications for when some symbol
All this does is request that the symbols subscribed to inform this
symbol of any changes, so it can then mark itself as having an invalidated cached
value (whether or not this is always true is dicult to say). An illustration of this
notication can be seen in Figure 4.4.
4.6 Simple JavaScript Agents
Each Symbol comes with methods for `observing' other symbols.
Currently, the
method for watching other symbols relies on the symbol table structure, as symbols
to observe are specied by a name in String form. This decision works ne in an
EDEN style setting, where every observable exists in a symbol table and is looked
up via a string, however it might have been preferable to observe symbols specied
by a direct reference to the symbol.
30
B is notified that it is no longer up to
date
B is A + 1
A
B is defined in terms of A
Spontaneous change in the value of A
A = 2;
Figure 4.4: Illustration of notication of change to a dependent symbol
31
watchA must inform any
observables it was previously
observing that it is no longer
observing them
watchA asks A to
schedule a call to
watchA when A
changes
A
watchA
watchA
remembers that
it is watching A
watchA spontaneously
starts watching A
Figure 4.5: An example of agent observation
It is actually possible for any symbol to observe other symbols, but it only really
makes sense to do so when the observer is some callable value (i.e. some JavaScript
function), because when the observed symbols are changed, a call to the value is
scheduled.
An example of how to do observation:
context.lookup(“watchA”).assign(function() {
alert(“A changed!”);
}).observe([“A”]);
Here, an action is stored under the name `watchA' and is set to trigger whenever
the observable named A changes. The way it does this is by requesting that the A
symbol schedules a call to it whenever its value is changed. Figure 4.5 illustrates the
interaction between the various symbols involved when one symbol is set to observe
another.
The dierence for notifying an observing symbol vs. notifying a dependent symbol is
that the notication to an observer is deferred until all of the dependencies have been
32
JavaScript VM
A call to watchA is scheduled for when
this transition has completed
A
watchA
watchA is observing A
Spontaneous change in the value of A
A = 2;
Figure 4.6: Illustration of scheduling a triggered action through
dealt with.
setTimeout
This is handled using the built in JavaScript function
setTimeout,
which allows you to ask the virtual machine to schedule a call to a given function
after a given timeout (or if no time is given, as soon as it gets a turn). This deferred
notication is illustrated in Figure 4.6.
4.7 Querying State
Each
Symbol
has a method for querying its current state (which is dened in the
shared prototype object). The method is:
Symbol.prototype.value
This method is quite simple - it checks whether the
Symbol
is up to date (each
symbol stores a ag to mark whether they have a faithful cached value), if it is, then
it just returns the cached value, otherwise it triggers a re-evaluation of its denition.
This has subtle but important connotations - if an observable is never inspected, and
it is dened through some formula (as opposed to having a value directly assigned
33
to it), but never has its state queried, it will never have its formula evaluated even if
its formula is dened in terms of symbols that are changing frequently. This should
do well in avoiding evaluating formulas too frequently.
4.8 Mutation of observables
As discussed previously, the trick for maintaining values by dependency is to ensure
that mutation of the managed state happens through an agreed interface. In this
case, each Symbol can be modied using helper methods:
• Symbol.prototype.assign
• Symbol.prototype.mutate
The implementation of these methods noties any dependent symbols, schedules
calls to of an update and removes any attached denition.
4.9 Note about Determining Dependencies Dynamically
Again, some care needs to be taken by the programmer about specication of which
symbols to watch. The approach to improving this situation implemented later is
to use a DSL which, when parsed, establishes the symbols used in the formula and
implicitly sets up subscriptions for you.
However, there is an alternative approach that has not been explored during this
project. Instead of determining dependent symbols at denition time, it could be
possible to have automatic deduction of the symbols to watch for cache invalidation
at denition evaluation time.
In this system, the context passed as the symbol
table would keep track of what symbols were looked up during the evaluation, and
implicitly subscribe the observable to these symbols (and only these symbols) when
the evaluation completed. This would have the benet of allowing the JavaScript
api to do automatic dependency detection, however it may be costly performance
wise, since in this system, whenever the formula is re-evaluated, it has to potentially
resetup dependencies.
34
5
EDEN Translation Scheme
This section attempts to document the various EDEN forms are translated to
JavaScript code while providing similar (unfortunately not always identical) meaning.
The aim is to provide a clearer guide than the code used for generating the
translator (which is basically a formal grammar with semantic actions), by using a
plain English description of the translation rules and examples. As well as illustrating the translation scheme, this section includes potential problems arising from the
translation itself.
5.1 Number, Character and String Data Types
The `primitive' EDEN data types include integers, oating point numbers, characters
and strings.
These were mapped directly to the closest corresponding JavaScript
data type.
JavaScript has less granular primitive types than EDEN. There is no true integer
data type - all numbers are stored using a 64bit oating point representation, though
there are some guarantees on operations that occur on values that t perfectly into
an integer representation[ECMA()].
JavaScript does however still have matching
notations for values with and without a fractional part, so number literals were
translated verbatim.
Further to this, there is no true `character' datatype in JavaScript, there are only
strings.
Strings however are sucient for representing a single character, just by
having a string of length 1, so all character literals are translated to string literals.
5.2 Observables
An observable in EDEN is denoted using letters, underscores and numbers.
A reference to an observable in EDEN source is translated to a call to the lookup
method on the object that eectively represents the EDEN symbol table from the
maintainer library. For example, a simple reference to an observable:
a
Is translated to:
context.lookup(‘a’)
35
How this is then used is dependent on the context in which the observable is in.
Essentially, the result of
context.lookup
(a
Symbol)
models an lvalue in the
EDEN notation. An lvalue is basically a location where something can be stored, and
so supports an assignment operation (for storing data) and also retrieving the data
stored there for use in more complex computations. The result of context.lookup is
Symbol.prototype.value
method (for retrieving the stored value) and Symbol.prototype.assign method
an object that supports these requirements through the
(for updating a stored value).
5.3 Arithmetic, String and Boolean Logic Expressions
EDEN supports a number of expressions for performing computations. Firstly there
are arithmetic expressions which can use various inx or prex operators, but also
there are expressions for comparison between numbers, boolean logic, and string
operations. Fortunately these are all very common within most programming languages, so there were operators in JavaScript that almost directly corresponded to
the EDEN ones. The translation scheme maps each operator to a native JavaScript
one where it can. For example:
a + b;
is translated to:
context.lookup(’a’).value() + context.lookup(’b’).value()
5.4 Dierences Between EDEN and JavaScript's Built In Operators
While these operators all seem close to the EDEN counterparts there are some dierences in their behaviour. Firstly, the addition operator in JavaScript is overloaded
so that it can deal with numbers and strings. In the string case, it performs concatenation of the operands, whereas in EDEN attempting to `add' strings is caught
at run time and reported as an error to the user.
“hello” + “world”;
type clash: number type required (got string)
while executing string near line 1, char 18:
36
To make matters even more confusing, JavaScript's type system allows for implicit
conversions of data types for operands in an expression. In particular, the `addition'
of a number and a string results in a string, where the number operand is coerced
to a string representation before concatenation:
2 + “world”
Another example where a dierence in behaviour is observed is with division.
In
EDEN, the division operator, when applied to integers, computes the result of integer
division of its operands.
However, since in JavaScript there is no dierentiation
between integer types and oating point types - all numbers have a oating point
representation. This means that in cases where the result of division has a remainder,
dierent results are seen from the EDEN expression to the JavaScript expression:
3 / 2 == 1 ## this holds in EDEN
3 / 2 == 1.5 // this holds in JavaScript
5.4.1 Pointers
EDEN supports referring to observables indirectly through what are called pointers.
In a sense pointers are `rst class lvalues', in that they allow some value representation of a location that can be passed around and manipulated.
In EDEN, the & symbol is used prexed to an observable to mean `location of '. Take
this as an example of storing a pointer to one observable in another observable:
x = 2;
ptr_x = &x;
Fortunately, objects created from the Symbol prototype eectively are rst class
lvalues, so we can store references to them in the observable symbol table or any
other lvalue we want:
context.lookup(‘x’).assign(2);
context.lookup(‘ptr_x’).assign(context.lookup(‘x’));
Accessing the location stored is done by using the * symbol prex to a pointer value.
The result of that expression is then a valid lvalue - so can be assigned to or used
to retrieve a value for a computation. E.g.
37
*ptr_x = 2;
y = *ptr_x + 1;
Would be translated to
context.lookup(‘ptr_x’).value().assign(2);
context.lookup(‘y’).assign(context.lookup(‘ptr_x’).value() + 1);
5.5 Lists
An example EDEN list literal:
[1, 2, 3]
Is simply translated to a JavaScript list literal:
[1, 2, 3]
5.5.1 List Indexing
In EDEN, it is possible to index an lvalue to retrieve another lvalue. This can then
be used to update and retrieve sections of EDEN list structures.
An important
property of these derived lvalues is that updates to the data stored for their location
expires their `parent' lvalue. For example:
list1 = [1,2,3];
list2 is list1;
writeln(list2); ## list2 == [1,2,3]
list1[1] = 5;
After the nal statement is executed, list2 needs to be notied that one of the terms
it depends on has changed.
The way this was modelled on the JavaScript side was by creating something that
had the same interface as Symbol, but with slight modications so that it would
update its parent symbol.
context.lookup(’list1’).get(1 - 1)
It was also necessary to adjust for the fact that EDEN uses 1-indexed arrays,
JavaScript uses 0 indexed arrays. This was dealt with by just substituting whatever
was used as the index with - 1.
38
5.5.2 List modication statements
JavaScript lists come with several methods that work well as substitutions for the
various list related statements in EDEN. The Symbol prototype supports a mutate
method - essentially a generalised version of
Symbol.prototype.assign - that
allows an update of a stored value directly using a given modication function.
Consider the following sample which inserts:
insert list, 1, 9000;
This is translated to:
context.lookup(‘x’).mutate(function(s) {
s.cached_value.splice(0, 0, 9000);
});
5.6 Denitions
It was important here to extract what the dependencies for the formula were so that
they could be watched for changes. This was implemented essentially by having a
boolean ag that represented whether the parser was looking at tokens that were in
a denition. Whenever an observable was encountered, the ag was checked to see
whether to make note of the observable (by adding it to a set).
The boolean ag was set to true when an `IS' token was seen by the parser, and set
to false when the the statement terminator was seen. This relied on the particular
parser algorithm used, which reads tokens from left to right, so will always see the
`IS' token before seeing all the observables in the denition.
5.7 Extracting Denitions in EDEN Form for Formula Inspection
In the TkEden environment it is possible to query observables for their formula
using a query statement, for example:
x is y + 20;
?x;
39
The query statement causes the following output:
x is y + 20; /* current value of x is @ */
x ~> []; /* x last changed by input */
So, this caused the need for associating each observable with a string which stored
the denition in EDEN notation. This was simply implemented by guring out the
span of the denition while parsing it, followed by extracting a substring from the
raw text being parsed.
The string was associated with the observable simply by adding a new property to
the Symbol object used to represent it (remembering that objects in JavaScript are
basically just associative arrays that can be extended at any time).
5.8 Functions
Dening a named function with the maintainer library is as simple as dening any
other observable, since functions are values in JavaScript. So some EDEN code like:
func blank { }
Is translated to:
context.lookup(‘blank’).define(function() {
return function() { };
});
Note that the denition actually evaluates to a function value.
This is the value
that will be looked up and called when the function is called.
5.9 Function Arguments
EDEN functions have access to a list of parameters in their body. Any symbol of
the form $N (where N is a number) refers to the Nth argument passed to an EDEN
function. For example, a simple square function can be implemented as follows:
func square { return $1 * $1; }
40
In JavaScript, the arguments keyword refers to a list like structure containing the
arguments passed in a function invocation, so is essentially synonymous to $ in
EDEN. Translating references to $ in EDEN to references to arguments in JavaScript
is a good start:
return arguments[0] * arguments[0];
However, in EDEN it is possible to perform assignment on the arguments list, that
is, $ is in fact an lvalue. Consider the following function:
func test {
$[1] = 1;
writeln(“$[1] = “ // str($[1]));
}
In test, the value of the rst argument is overridden with the number 1, and then
printed. As a testcase:
test(9000);
Running this causes the output $1 = 1, showing that the argument has been
successfully overridden. The implications of this are that we need references to $
to be translated to some Symbol, which is what models an EDEN lvalue in the
JavaScript system. Two steps were made:
1) Every translated function also comes with a line converting the arguments array
to a Symbol so that it could support `assignment' in our system, for example, the
blank function in section 5.8 would be translated to:
context.lookup(’blank’).define(function() {
return function() {
var args = new Symbol(asJavaScriptArray(arguments))
};
})
There is one small thing to notice: The arguments keyword doesn't actually refer
to a full-edged JavaScript array (in that it doesn't support the methods dened
in the Array prototype object). Since the translation scheme relies on EDEN lists
being translated to JavaScript arrays,
Array.prototype
asJavaScriptArray
methods to the arguments structure.
41
is used to add the
2) References to $ are now translated to references to the local variable `args', this
local can then be treated similarly to any other lvalue.
For example, assignment
looks much like it did with assignment to observables:
args.assign(10);
Or in order to assign to a particular position in the arguments:
args.get(0).assign(10);
5.10 Function Argument Aliases
EDEN has a syntactic sugar for referring to function arguments by a name instead of
just by position. These are declared at the start of a function using a para statement,
like so:
func square {
para number;
return number * number;
}
In the current translator, these para statements are not really translated to JavaScript
code. Instead, the parser just makes note of the para aliases at the start of the function, and if it encounters any of those aliases later in the function body, it translates
those references to the corresponding argument position for the alias.
The above
code would be translated to:
context.lookup(‘square’).define(function(context) {
return function() {
return args.value(0) * args.value(0);
};
});
42
5.11 Function Local Variables
EDEN supports `auto' variables which are scoped to within a function denition.
JavaScript also supports declaring function local variables through use of the var
keyword, so a corresponding function local was created in the generated JavaScript
code for each function local. For example:
func sumToN {
para n;
auto i, sum;
sum = 0;
for (i = 1; i < n; i = i + 1) {
sum = sum + i;
}
}
Is translated to:
function() {
var local_i = new Symbol();
for (
local_i.assign(1);
local_i.value() < args.get(0).value();
local_i.assign(local_i.value() + 1)
) {
local_sum.assign(local_sum.value() + 1);
}
}
One nice property of function local variables with this implementation is that they
don't require performing a hashing operation in order to retrieve a location for
storing and retrieving values like regular observables currently do.
This could be
important for making certain types of computations more ecient (e.g. any computations where a single location is updated or read lots of times).
43
5.12 Triggered Actions
Triggered actions are modelled on the JavaScript side in the same way as EDEN
functions, by just dening an observable as some function value that can later be
called. For example:
proc WatchA : A {
writeln(“a changed”);
}
Is translated to:
context.lookup(’WatchA’).define(function(context) {
return function() {
context.lookup("writeln").value()("a changed");
};
}).observe(["a"]);
5.13 Control Flow
This was the most straight forward part of the translator to implement, since the
JavaScript control structures resemble the EDEN ones very closely. The structures
EDEN supports are:
•
If statements
•
Switch statements
•
While loops
•
C style for loops
•
Break/Continue
All these are almost directly transferable to JavaScript. For example, a for loop in
EDEN looks like:
for (i = 1; i <= 10; ++i) { ## code ... }
Would be translated to the following JavaScript:
44
for (
context.lookup(‘i’).assign(1);
context.lookup(‘i’).value() <= 10;
context.lookup(‘i’).assign(context.lookup(‘i’).value() + 1)
) {
// code ...
}
The only dierence is in how the value of
i
is retrieved.
5.14 Verbatim JavaScript Sections
An extension was made to the EDEN syntax which allows for marking sections of
code to be translated to JavaScript with no modications. In order to mark such
a section, the code needs to be surrounded with `${{` and `}}$'. The intention of
this feature was to allow for the creation of agents that updated some state in a
visualisation, e.g. some HTML elements in a page. Take this example of an agent
which alerts:
proc WatchA : A { ${{ alert(“A changed!”) }}$; }
This generates the following JavaScript:
context.lookup(’WatchA’).define(function(context) {
return function() {
alert("A changed!");
};
}).observe(["A"]);
5.15 Methodology for Developing the Translator
The Jison parser generator was used to create the program that translated EDEN
source into JavaScript source. Since Jison itself is written in JavaScript, it was simple
to develop the grammar le in the web browser and interactively test translating
and evaluating examples. The interface used in this process is shown in Figure 5.1.
45
Translator Grammar
Generate Translator
Test Input/Output
Figure 5.1: Screenshot of the grammar testing and development environment
46
Figure 6.1: Prototype JsEden Interface
6
Example Applications
6.1 JsEden Prototype
A prototype interface for what JsEden might look like was developed using a combination of maintainer.js, the EDEN translator and some simple HTML/JavaScript
for the user interface. The nal interface is shown in Figure 6.1.
The features it includes are:
1. An interpreter for EDEN statements
2. Simple error reporting
3. A naive visual symbol table, allowing for in place modication of model state
6.2 General Approach to Developing Visualisations with JsEden
Models
Usually it is desirable to develop some visual representation of what is trying to be
modelled. These can be implemented using triggered actions that update some state
47
in the webpage that JsEden runs in, whenever certain parts of the underlying model
state change. However, before developing a model that uses any sort of visualisation,
some care needs to be taken to ensure that any required preconditions in the state
used for the visualisation have been fullled (for example, initialising some graphics
system, adding some element to an HTML page which will then be drawn within).
In a TkEden model, all the visualisation state is described in a denitive notation.
Of course, not all the state for a DoNaLD picture is explicitly described (things
like the algorithms used to draw lines, all the state of the tcl/tk toolkit used),
however, due to the engineering of TkEden, any `external' state required for the
correct visualisation is guaranteed to be initialised on TkEden starting up.
In the JsEden environment, there is currently no state setup on startup for line
drawings. Furthermore, there is no ability to use the DoNaLD notation to describe
line drawings. However, DoNaLD is not too important as it is simply some sugar
around already understood EDEN statements. This can be veried by using TkEden, which allows for inspection of translated DoNaLD denitions. For example the
following donald code:
line myLine
myLine = [{0, 0}, {100, 100}]
Results in the following EDEN being generated:
_ is [ OPENSHAPE, &_myLine ];
_myLine is line(cart(0, 0), cart(100, 100));
A_myLine = "";
proc P_myLine : _myLine, A_myLine, DoNaLD {
plot_line(DoNaLD, &_myLine, &A_myLine);
}
Here what has happened is that the line `declaration' is essentially a macro which
generates a denition for an agent which is responsible for updating the external
visualisation state.
What we can do in the JsEden environment is to use raw JavaScript to setup what is
needed before constructing a visualisation. Subsection 6.3 demonstrates an example
of doing this by wrapping some JavaScript in EDEN functions, which construct and
manipulate an HTML canvas element (which is for general puprose line drawing).
48
6.3 Example JsEden Model With A Simple Box Visualisation
The following listing shows an EDEN script which uses HTML canvas for drawing a
box. Any changes to the
boxX, boxY, boxW or boxH observables causes an update
in the visualisation. This example is available for testing as a sample in the grammar
development page (along with a few others).
##
## HTML canvas utilities
##
## sets up some necessary "external" state
proc setupCanvas { ${{
$(’body’).append(’<canvas id="c" width="400" height="400"></canvas>’);
}}$; }
proc clearCanvas { ${{
var c = $(’#c’).get(0);
c.width = c.width;
}}$; }
proc drawBox { ${{
var c = $(’#c’).get(0).getContext(’2d’);
c.fillRect(arguments[0], arguments[1], arguments[2], arguments[3]);
}}$; }
setupCanvas();
##
## example agent that draws a box based on some observables
##
proc boxDrawer : boxX,boxY,boxW,boxH {
clearCanvas();
drawBox(boxX, boxY, boxW, boxH);
}
boxX
boxY
boxW
boxH
=
=
=
=
0;
0;
20;
30;
The result is shown in Figure 6.2.
49
Figure 6.2: A simple visualisation using a square
6.4 Embedding Formulas Into an HTML Document
The idea in this example is that a user could enter a formula inlined in some HTML
markup like so:
<formula>x + 1</formula>
And this tag would have its contents replaced with the contents of the given expression. Here was the markup used in this example:
<h1>dependency demo</h1>
<p>the current value of x + 1 is <formula>x + 1</formula></p>
<p>the current value of x + 2 is <formula>x + 2</formula></p>
<p>feel free to modify x here: <input type="text" id="xInput"/></p>
Figure 6.3 shows the resulting webpage.
The way this was implemented was quite simple. Once the page has nished loading,
some JavaScript executes which scans the page for
following for each:
50
formula
tags, and creates the
Figure 6.3: Web page with embedded formulas that reect the maintainer.js state
•
An observable (called
formulaElementN,
where N corresponds to the Nth
formula tag) which is dened using the expression inside (interpreted as EDEN).
•
An agent is created which watches this observable for changes, and propogates
the change to the HTML.
This example can be found in the project source distribution as
51
document-maintainer.html.
7
Pro ject Management
This section gives an overview of when the work required for this project happened,
as well as what approach was taken to managing the work involved. As well as this,
it gives a rough idea of other issues related to getting the project done, such as the
tools used to do so and any external events that might have aected the work.
7.1 Timeline
The MSc Computer Science and Applications course for which this project was carried out for started at the beginning of October 2010. Students are required to select
a supervisor and project by the end of the rst term, as well as submit a specication for their project idea including analysis and background detail. Professor Steve
Russ kindly agreed to supervise for this project.
The way this project has turned out is not very close to the original plans made
during this rst term. The project supervisor has stayed the same, and the overall
eld of modelling with denitive scripts has stayed the same, but the original focus
was on a currently existing tool called Cadence[Pope(2011)].
In some respects the project had similar ideas in mind - Cadence (and the main
denitive notation used to interface with it, DASM) while interesting from both a
modelling perspective and a software development perspective, is currently a little
bit lacking in accessibility as both the complex underlying maintainer and the notation itself are somewhat undocumented through literature and examples. In the end
this is one of the main motivations of JsEden - to create something very accessible
to the user and the developer, via the use of web technologies to allow it to work
across platform with no setup as well as oering a robust development stack.
This is a project that I have had in my head for a while, so it is dicult to document all the time spent towards it.
However, it is denitely true that the bulk
of the work developing the maintainer and the translator happened in a few weeks
during February/May. The report documenting this was written much later, during
July/August/September.
7.2 Methodology
The methodology for the work done for this project was quite straightforward. The
requirements for the project did not really change during the course of it, it was
52
simply a case of attempting to achieve goals in order and assessing while the process
was continuing.
There were a few cases where the maintainer being developed
parallel to the translator would need to be modied to make the translator neater.
Testing was carried out during the development of the parser to attempt to prevent
regressions from occurring, as new features of the EDEN language were implemented, test cases were created for them.
The test cases currently in place have
been documented in section 10.2.
7.3 Other commitments
While working on this project I have also been involved in some part time work
doing software development (also involving JavaScript and rich interfaces).
This
work started in early March and has varied in the amount of time committed per
week - ranging from 3 hours to 30.
As a result of this there has been a section of my time and eort that could not go
to this project, however since all my other modules were completed last year, I feel I
have managed to dedicate a decent time to both projects and have no regrets about
my decision to do both.
7.4 Version Control and Backups
Throughout the project the git version control system was used for managing the
codebase. This is an advanced system for storing snapshots of your codebase which
in git are called commits. This system proved useful in several ways:
•
Easy tracking of current changes to the project by generating a `di ' which
shows all edits/additions/removals from the last stored snapshot.
This was
useful for both guring out what elements had changed when something was
behaving in an unexpected way, and also for remembering what I was doing
the previous day.
•
Reverting the codebase back to a previous snapshot in the case of a problem.
•
A log of all the snapshots including both a di and a message from the author
to describe the changes introduced between that commit and the previous one.
•
Simple (and ecient) synchronization of the codebase including the entire
commit history to remote locations, making for eective backups.
53
•
Branching o from a particular stored state so that it was possible to work on
multiple trains of thought if I wanted to switch between features that would
required large changes to the codebase, as well as semi-automated merging of
codebase states.
7.5 Tools
The toolchain for the software development process was quite simple.
The key
elements of it were:
VIM[Moolenaar(2011)] which is a simple programmer's text editor, available for
various platforms including Windows, GNU/Linux and Mac OSX.
For testing, various browsers were used as there are occasional subtle dierences
in their behaviour (usually in situations where the DOM/JavaScript specication
aren't specic). The browsers tested in were:
•
Google Chrome
•
Safari
•
Firefox
•
Internet Explorer 8
•
Mobile Safari (for iPhone)
Further to this the Firebug extension for Firefox browser was useful during the development process. It is an extension that provides sophisticated debugging facilities
for JavaScript development including an interactive console, inspection of JavaScript
data structures and reporting of JavaScript errors with the ability to jump to the
problematic location in the code. This proved invaluable for both debugging problems and for experimentation with both JsEden and the core maintainer library.
7.6 Libraries
Several JavaScript libraries were used to aid the development of JsEden:
•
jison - http://zaach.github.com/jison/docs/ - a parser generator used to develop the translator for EDEN to JavaScript
54
•
jQuery - http://jquery.com/ - a general purpose DOM manipulation library
used in the UI
•
jQuery UI - http://jqueryui.com/ - a jQuery plugin that adds some common
UI widgets
•
Douglas Crockford's implementation of JSON parsing/serialisation in JavaScript
- http://www.json.org/js.html - for browsers which don't have the JSON object implemented natively
•
jsbeautier - http://jsbeautier.org/ - a script to tidy up JavaScript whitespace/indentation, to format the outputted JavaScript from the translator for
easier inspection
7.7 Legal, social, ethical and professional issues
Generally this project did not touch on many legal, social, ethical and professional
issues.
The main things to consider (for any software product) were related to
licensing of the code written itself and any libraries used in it's development.
The code written by the author includes a copyright notice as well as a notication
that it is all licensed under the MIT Licence[OSI()], which is a very permissive
license in that it only requires a copy of the license is provided with the code or any
derivative work.
The other code used is all library code, all of which are licensed under a permissive
license:
•
jQuery - MIT
•
jQuery UI - MIT
•
jison - MIT
•
Douglas Crockford's json2.js - Public Domain
•
jsbeautier - Public Domain
55
8
Conclusions
8.1 Success of targetting EDEN
The JsEden prototype could be a great asset to the Empirical Modelling community.
The EDEN translator developed to help reuse EM models is reasonably comprehensive but still needs more testing, as well as `builtin' functions being ported over (in
some cases this is just including functions written in EDEN in the JsEden environment). While it oers something similar to WebEden, the architecture is much
simpler and the ability for responsive interactions is far greater.
As for the other tools available for interactive modelling, it potentially oers a clean
break from TkEden which has become stale.
The codebase is large and over the
years has been patched by many people without too much code review to keep it
manageable. This makes it far less accessible for new developers than the JsEden
architecture which only requires a very small amount of code written outside of
denitive scripts.
The following is a small summary of some of the advantages of JsEden over TkEden
and WebEden:
•
JavaScript is more accessible than C (which the TkEden VM is written in), as it
already has high level language features such as automated garbage collection,
resizable data structures and rst class functions. This is particularly relevant
as EDEN is mostly of interest in the University of Warwick department of
computer science - where there is basically no ecosystem for C developers.
•
The JsEden Codebase is (currently) much smaller than the TkEden one. The
JavaScript maintainer is less than 350 source lines of code, the translator is
about 500. The EDEN virtual machine and language interpreter alone are over
10,000 lines of code. While this might not be a particularly accurate measure
of code complexity, the scale of the dierence here should illustrate there is
certainly a signicant delta.
•
Easily extensible by embedding JavaScript code into EDEN code to hook into
rich browser APIs. This means that it is much easier to extend or modify the
tool within the tool itself.
•
Guaranteed to work for a very long time as JavaScript is too widespread to be
deprecated any time soon.
56
•
Simple architecture making it trivial to deploy or migrate. No need for clever
load balancing or authentication beyond dealing with HTTP trac to get the
requisite JavaScript les.
•
Much easier to scale than WebEden, which required an instance of TkEden
somewhere serverside for each instance of WebEden.
•
JIT compilation for denitions, whereas in TkEden, the result of a denition
is computed using the hand written interpreter.
•
Competition in JavaScript engines drives development to improve performance.
Saying this, obviously a lot of work still needs to be done for this platform to act
as a substitute for TkEden, which has had numerous features implemented over the
years.
8.2 Potential outside of EDEN
While the EDEN language provides a great DSL that makes the use of dependency
a lot less error prone, the JavaScript API oers a lot more exibility in what sorts
of things can be connected together. The main problem is that EDEN can't quite
understand everything that exists in the JavaScript world, though this might be
dealt with by making some minor extensions to the EDEN syntax.
JavaScript already has an event based programming model, which allows for easy updates of visualisations in the event of changing data, so the real gains of maintainer.js
would be seen in easier maintenance of program invariants through dependency.
57
9
Further Work
This section documents some ideas for interesting extensions on the work done in
this project.
9.1 More Model Development
In order for a tool to be useful it has to be used. The maintainer is already in a state
where models can be developed interactively, however without constructing larger
scale models it is dicult to assess how successfully it can be applied.
9.2 Implement Cyclic Dependency Checking
Currently maintainer.js does not check for attempts to create cyclic dependencies
between symbols. This can cause an innite loop when expiring a symbol's cached
value. This could be implemented quite simply by adding some code to the symbol
subscription method to check for cycles before commiting the change.
9.3 Possibilities outside the browser
While JavaScript is mostly executed in a browser environment, it is also possible to
execute it in a less strictly controlled environment (mostly this is used for serverside
web programs).
One example of such an environment is called Node.js, which combines Google's
v8 JavaScript engine with libraries for enabling it for general purpose programming
outside of the browser. One benet of running the maintainer via Node.js instead
of a browser based environment is the simplicity of interfacing it with other computer artifacts, for example using its simple Foreign Function Interface (FFI) to
provide action to C apis[Branson()] or by using other inter process communication
like named pipes or sockets.
This could open up possibilities of using more so-
phisticated graphics engines or interacting with other tools or peripherals. Node.js
is targeted at communicating systems which could allow it to shine as a sort of
glue with the ability for maintenance via denitions, not unlike some of the uses
of Cadence.
Another environment for using JavaScript outside of the browser is Rhino, which is
a JavaScript engine that runs on the Java Virtual Machine (JVM). The JVM has
58
recently gained improved support for dynamically typed languages (mostly due to
a rising popularity in the implementation of the Ruby language for the JVM, which
is dynamically typed)[Oracle()]. JavaScript run through Rhino has access to Java
APIs available on the system, so this could open up some interesting possibilities
for interacting with already existing Java programs or user interfaces from triggered
actions.
9.4 Collaborative modelling and interaction with already existing tools
There are possibilities for remote communication between instances of JsEden in either the browser setting or one of the headless settings. The communication methods
available in the browser are:
•
Comet
•
EventSource
•
WebSockets
Sadly all of these models follow a client-server model, where the browser is always
a client.
In other words, there is no possibility for peer to peer communication
between browsers (at least when using JavaScript, it may be possible with Flash
or Java which has more capability to elevate based on user specied permissions).
This means there would always be the requirement for a high bandwidth machine
that has to proxy the communications between browsers.
Things are easier outside of a browser setting, where JavaScript is less limited as the
program is trusted in this environment. Node.js has great support for networking
over TCP or UDP, so could be used along with the JavaScript maintainer to produce
a collaborative modelling environment.
It may also be possible to have collaboration between a JsEden based environment
and already existing modelling tools such as dtkeden (distributed TkEden) or Cadence which already have well dened protocols for distributed formula maintenance.
9.5 Modelling with hierarchical structures
Many programming languages allow for modelling problems using objects, which
are essentially a way of modelling ownership of certain state (as well as sometimes
enforcing where changes to such state can come from).
59
JavaScript already supports object oriented modelling, however the project has
mostly ignored the possibility of using such ownership relationships in the dependency enabled environment.
Notably, this is topical in the Empirical Modelling
community of late due to the more recently developed Cadence tool, which is heavily focused on hierarchical graph structures for its state.
These sorts of structures should also be usable in a way similar to `collections' used
in JavaScript data binding oriented frameworks, which allow for setting triggered
actions that watch for elements being added or removed to the collection. Currently
JsEden (and TkEden) make it dicult to create triggered actions that watch for the
introduction of (or removal of ) new symbols before acting.
This capability could
be particularly useful is for the creation an ecient visualisations of a collection
which can vary in size dynamically. One example of a scenario like this is the visual
symbol table' which shows the denitions for all the observables currently dened.
9.6 Support for other notations
The vast majority of the interesting models developed using TkEden use some kind
of graphical interface. This was generally developed using a separate notation for
graphics, such DoNaLD, SCOUT or the gel notation. While the use of multiple notations might not be the best approach for dealing with multiple concerns available,
the support for these notations would help support the reuse of the EM archive
projects.
In scripts written for TkEden it is possible to switch what notation is being used
using a directive such as:
%donald
It would be quite simple to add support for multiple notations by creating a higher
level translator which dispatches translation o to a separate translator function for
the currently active notation, and switches which translator to dispatch to whenever
a
%donald
style comment is seen.
9.7 Improved JsEden UI
Some more work on the UI could make the modelling process more accessible and
raise the perceived quality of the software. Improved support for code editing - there
60
are a few JavaScript/HTML editors that could be used instead of just an HTML
<textarea>
for code editing. These would allow for syntax highlighting and even
syntax error highlighting. An example of a library that already supports better code
editing is CodeMirror[Haverbeke()].
9.8 Improved Error Reporting for EDEN Code
Currently the error reporting for the EDEN interface to the JavaScript maintainer
is limited to syntax errors, however it is also possible for there to be run time errors
in the JavaScript generated by the EDEN translator. JavaScript will already report
these errors to the browser's error window, but in this form they don't aid the use
of the EDEN environment as much as they might for the following reasons: They
make little sense in the context of EDEN as they actually refer to the generated
JavaScript code. Since the errors occured in dynamically generated and evaluated
code, they have no source attachment to point to.
It may be possible to deal with this by using the attached denitions of functions,
however guring out how line numbers correlate between the generated JavaScript
and the original EDEN source will be challenging.
9.9 Persistency for the developed models in JsEden
The prototype JsEden environment does not store any of the work done between
page loads. There are several useful features that could come from work to implement this:
•
Inspecting history of interactions for educational reasons
•
Collaboration, by transmitting the model to other people
•
Taking breaks from working on a model
•
Returning to a previous model state, possibly storing multiple branches of
models developed from some particular state
In order to persist models, the system could either:
1. Store a snapshot of the current model state
2. Store a history of all the interactions during the model development.
61
Both approaches are interesting and have their own considerations for implementation and use cases.
The rst approach is ideal for restoring a model to a previous state quickly. It does
rely on the current state being completely serialisable to something like a string in
some way, which is ne if the model state is completely representable in terms of
some denitive notation.
The ability for sections of script to be translated verbatim to JavaScript in the
EDEN like language of JsEden introduces problems here, as the model can become
reliant on some `external state' which is not seen as a simple observable in the
JsEden environment. It is quite easy for this to happen when dealing with function
values which rely on some state which is in their closure (this is a common idiom
for encapsulating state for event callbacks in JavaScript) but not accessible as an
observable. Another possibility is that some JavaScript code relies on the state of
the HTML page which it has access to. This sort of situation makes it impossible
to take a faithful `snapshot' of the model, as there is no way to enumerate all of this
state.
The second approach is necessary if users wish to gain insight into the interactions
performed in the model development, but really it is impossible to capture all
interactions - trying to capture all interactions (which potentially includes all mouse
interaction) would be unreasonable, so it would be limited to capturing interactions
through the EDEN interface. This would mean that in order to restore a model, it
really is necessary to be able to enumerate all the model state as observables.
It is also important to consider where exactly information about model development
should be stored. The possible solutions are:
1. Store everything locally using new HTML5 APIs
2. Store things on a remote machine by using POST or REST requests to some
server side app
Option 1 has the benet of requiring no server side application in order to implement persisting developed models. It also allows the user to do model development
without the need for an Internet connection. The local storage APIs introduced in
HTML5 are reported to work in most modern browsers [Grove()], and due to the
introspective nature of JavaScript, it is possible to do feature detection and fall back
on some 3rd party system for persisting data on the client side (for example Google
Gears[Google()]).
62
As for Option 2, one good example of how to do this is jsddle.net.
This is a system that allows you to develop web applications within the browser,
and store them persistently on their servers. You are they given a url which you
can use to return to that particular version of the webapp. The format is always
http://jsddle.net/<unique-id>/<revision-id>
63
10
Appendices
10.1 Build Instructions
HTML/JavaScript applications generally only need a JavaScript enabled browser
in order to run.
The architecture for the JsEden environment is currently quite
simplistic, and in order to host your own instance of it, all that is required is to
extract the source distribution and open eden.html in a browser. Making it public
is simply a case of putting it somewhere being served up by a webserver.
If you're using a machine in the department of computer science at Warwick University, you can run (this will also work on most GNU/Linux or OSX machines):
curl "http://trmonks.me.uk/gitw/?p=js-dependency.git;a=snapshot;h=refs/heads/\
master;sf=tgz" | tar xzvf
The grammar development and testing environment is test.html, the prototype
JsEden interface is eden.html.
10.2 Test Cases
Some testcases were created for the translator during it's development. The tests
are all in js/tests.js in the JsEden distribution, here is a summary of what is tested
for currently:
1. Lookup results in a symbol
2. Assignment sets the correct value
3. Increment results in a value 1 greater
4. += op works
5. The value after formula denition is correct
6. Function denition stores a function in the symbol table
7. Procedure denition stores a function in the symbol table
8. Triggered action denition stores a function in the symbol table
9. Triggered action denition observes a requested symbol
10. Function calls work Return statement from a function works
64
11. Function parameters work
12. Parameter aliases work
13. Nested strings work
14. Multiline strings work
15. Multiple function denitions with locals works
16. Underscores in observable names works
17. Autos work in a function denition
18. Autos work in a procedure denition
19. Autos protect the outside scope
20. Single quoting a character works
21. Empty quotes won't parse
22. Single quoting an escaped quote character works
23. Lists are 1 indexed
24. .get(i) returns an accessor with index i
25. .get() returns an lvalue
10.3 Glossary of Technical Terms
Closure - The closure of a function is the set of all things in its scope.
XML - A standardised markup for representing arbitrary data, along with some
schema.
JSON - JavaScript Object Notation.
This is a simple string representation of
JavaScript objects for communication.
DOM - Document Object Model.
This is a JavaScript object representation of
HTML documents to allow for simple manipulation (without parsing HTML)
DSL - Domain Specic Language
EDSL - Embedded Domain Specic Language. This usually refers to a system of
values in an already existing programming language that resemble a DSL, often with
help from the type system to enforce language rules.
VM - Virtual Machine
65
References
[Ashkenas(2011)] Jeremy Ashkenas.
URL
The coeescript project page, August 2011.
http://jashkenas.github.com/coffee-script/.
[Beynon(2006)] Meurig Beynon. Syntax summary for eden, November 2006. URL
http://www2.warwick.ac.uk/fac/sci/dcs/research/em/software/eden
/langref/summary/.
[Branson()] Rick
Branson.
A
node
tutorial.
URL
https://github.com/rbranson/node-ffi/wiki/Node-FFI-Tutorial.
[ECMA()] ECMA.
The
ecmascript
language
specication.
URL
http://www.ecma-international.org/publications
/files/ECMA-ST/Ecma-262.pdf.
[Google()] Google.
The
google
gears
project
page.
URL
http://gears.google.com/.
[Google(2000)] Google.
The
v8
project
page,
January
2000.
URL
http://code.google.com/p/v8/.
[Grove()] Ryan
client-side
Grove.
Yahoo!
storage
to
improve
search
user
pad:
Using
experience.
URL
http://wonko.com/post/search-pad-browser-storage.
[Haverbeke()] Marijn
Haverbeke.
The
codemirror
project
page.
URL
http://codemirror.net/.
[Moolenaar(2011)] Bram Moolenaar.
The vim project page, April 2011.
URL
http://www.vim.org/.
[Oracle()] Oracle.
TM platform.
java
Jsr
292,
URL
supporting
dynamically
typed
languages
on
the
http://jcp.org/en/jsr/detail?id=292.
[OSI()] OSI. The mit software license. URL
[Pope(2011)] Nicholas Pope.
http://www.vim.org/.
The cadence project page,
August 2011.
URL
http://www2.warwick.ac.uk/fac/sci/dcs/research/em/software/cadence/.
66