>> Nigel Hess: So my name is Nigel Hess, and I'm an account executive with
Wolfram. My background is in mechanical engineering. So I got my
undergraduate from Purdue University and got involved in kind of account
management and technical sales with Wolfram.
I have with me Michael Kelly. I'll wait till Michael comes up here to kind
of introduce himself and talk a little bit.
But I thought I would start off with giving sort of a higher level technology
overview. I'd like to talk a little bit about what the Wolfram Language is
and kind of what the philosophy is behind it and kind of the point of it.
And then start talking about the different ways of leveraging the language
and making use of it, so from everything from kind of using it right in front
of you to using it to create tools and interactive applications and even
reports and different ways of kind of pushing all this stuff out to other
people.
So I will go ahead and get started with this overview for about ten minutes,
and then I'll hand it off to Michael to get into some more specific
application areas and stuff like that.
So the language at the heart of Mathematica is now kind of formally called
the Wolfram Language. And it's a language that we basically incubated in
Mathematica for the past 25-plus years. And it's a language that attempts to
incorporate and build in as much knowledge as possible.
So both knowledge in the form of various datasets, which are accessible
through Wolfram|Alpha, but also a great deal of knowledge about computations,
whether it's in domains like machine learning or statistical analysis or
image processing. All of these different domains we've really built into the
core language.
So I won't spend too much time on these tabs, but these are a lot of the kind
of objectives of the language. So building in a great deal of automation to
make kind of things quicker and easier for you to build as somebody
developing content.
And of course as Michael will probably show you, you can go into various
functions to set things up specifically as you desire to. So everything
doesn't need to be very high level and automated.
So deployment is really an area we've focused on as well in the last couple
years. So where we used to -- whoops, there goes my mic. So where we used
to really be talking about Mathematica and what you could do with
Mathematica, now the shift -- we've kind of shifting to talking about that
core language and that core technology and the different ways you can push
that code out and make it usable to not only you but your colleagues and
managers and clients and customers as well. So that's kind of the point of
this slide.
And I mentioned built-in knowledge. So Wolfram, the Wolfram Language is
integrated with Wolfram|Alpha. And Wolfram|Alpha has access to literally
trillions of data points, everything from weather data to data on chemicals
and compounds to stock data, and all of that data is available to you and
easily accessible in the Wolfram Language.
So this just kind of demonstrates a couple of the domains or different areas
that are really touched on by the Wolfram Language. So there are functions
in the language that are specific to doing things related to image
processing. There are functions related to machine learning, to statistical
analysis, to creating visualizations.
And all of it is available to you in this kind of one consistent environment,
consistent framework.
So these are also a couple of the application areas that Michael will get
more into when he comes up and speaks.
So Wolfram Language is actually used on the back end of quite a few
technologies, some of which you might use. So if you have an iPhone or a
Galaxy phone and you use technology like Siri, when you ask certain
questions, our technology -- like Wolfram|Alpha is actually being used on the
back end to find the answer and stuff like that.
But we're also started to embed the language only devices. So if you buy a
Raspberry Pi, for example, today, it actually comes with the Wolfram Language
embedded on that device, can which is a nice way to kind of quickly tinker
around and play with it and stuff.
So this just kind of shows the continuing development of the -- all of the
functions that are built into the Wolfram Language. So you can kind of see
that it's really continuing to expand. And a lot of that is because we've
taken these various application area which used to require purchasing an
add-on or a package or something like that and we've now built them into the
core language.
So now if you want to do things in image processing, for example, all that
functionality is built into the language. If you want to do machine
learning, that's built in too. Which is nice because you don't have to worry
about purchasing add-ons and extensions and stuff like that.
So this slide just talks a little bit about how kind of fine grained some of
these functions in Mathematica can get. So there's thousands of functions,
over 4,000 functions now. And you can really get down into pretty
nitty-gritty specific areas.
So Mathematica and the Wolfram Language can also support hundreds of
different file types. So more than likely, if you're looking to get that
data into the Wolfram Language or Mathematica or get it out, more than likely
you can.
And the Wolfram Language is also able to integrate with all of these -- or
connect to all of these other languages and platforms too. So that's
something good to keep in mind as well.
So I think historically Mathematica and the language have really been well
known for creating very aesthetic and beautiful visualizations. So I thought
I would take a quick minute just to show some of those visualizations, just
to give you a good idea of the full spectrum of visualizations and charts and
plots that you can create in Mathematica.
So this gives you a good sampling of 2D visualizations. So plots, contour
plot, polar plots, and so forth. We have all of the 3D visualizations
covered as well. So this is kind of a little bit of sampling of that too.
So of course if you want to do charting, we have all of that, from some of
the simpler stuff that you might associate with Excel to more complex,
sophisticated stuff like 3D bubble charts.
And of course statistical analysis is also a very major area in the language.
So we've got all sorts of different charting and visualizations around
statistical analysis as well.
So one of the great things about Mathematica is its built-in notebook
interface. So when you open Mathematica up, you have this kind of palette
called a notebook. And what's great is that you can put just about anything
in a notebook. So it can contain all of your formatted code and everything,
as well as maybe comments on your code. You can see in here you can have
images, you can have formatted text and sections.
And we'll talk about a little bit later your notebooks can even contain
dynamically interactive content. So it might be a little bit difficult to
see here, but we'll show this a little bit later, but you can create very
easily these sort of GUI objects with sliders and drop-downs and all sorts of
interactive elements that when you make adjustments to them, they actually
change kind of underlying parameters of the model.
So that can be a great way of building tools that you want to share with your
colleagues or maybe you want to give your manager a model that allows him to
vary some parameters and gain some insight into that model. This makes it
really easy to do that.
So, in fact, this presentation is even given in a Mathematica notebook. So
this is a Mathematica notebook right here. So to prove that, you could kind
of just click down here and do something like that and kind of just see that
this is a live notebook.
So I put this slide together just to make it clear that Mathematica and the
Wolfram Language at the heart of Mathematica, you don't need to just use it
as somebody just sitting in front of the desktop. Mathematica has built-in
functions like schedule task that allow you to put Mathematica up on a server
or something, and it could be the back-end computation engine for some other
program that you're writing or it can do all sorts of automated things
without you having to necessarily be right in front of the computer.
So I'll just touch on these briefly. GridMathematica is basically a package
of dedicated Mathematica kernels. So if you're doing a lot of work related
to high-performance computing or parallelization, gridMathematica can be
quite useful.
So one thing you could do with it is if you have a computing cluster or
something like that and you have gridMathematica, you can actually send a
computation from your Mathematica front end to gridMathematica and have it
parallelized across many CPU cores and then have the results sent back to
you.
So in that way you can really take advantage of other hardware that you may
have on the network if, say, you know you're using a limited laptop or
something like that.
And webMathematica I'll just very briefly touch on. But webMathematica is
really about putting Mathematica, kind of the power of Mathematica on the
back end of a Web service. So I won't go into it too much right now because
I don't think it relates probably to most people's project, but if that is a
curiosity, certainly let us know. We'll put our contact information up at
the end. And we can certainly explore these topics further with you.
So most of you are probably familiar with Wolfram|Alpha. Wolfram|Alpha is
our computational knowledge engine. And it's also a great resource for
pulling all sorts of different day into the Mathematica environment. So as I
mentioned, there's weather data, there's stock data, there's data on chemical
compounds, really just a ton of data, a lot of which could be of use to your
projects, depending on what you're doing.
So it's nice to know that Mathematica is integrated with Wolfram|Alpha so you
can easily pull all of that data and all of those resources directly into
Mathematica.
So this schematic kind of just talks a little bit about how Wolfram|Alpha
actually works. I won't really go into detail on it. But if anybody is kind
of curious about that, we can certainly talk more about that.
So while we have Wolfram|Alpha publically available at wolframalpha.com,
which is also accessible through your local copy of Mathematica, many
companies have also gotten a private instance of Wolfram|Alpha. And one of
the great things about Wolfram|Alpha is it allows you to ask natural language
questions on datasets essentially, and then get back not only like a
customized -- not only a precise answer, if it's available, but also can kind
of customized report of other things that it thinks might be also of interest
to you based on the question that you asked.
So if you wanted to take this sort of natural language-asking technology and
apply it to your datasets, the Wolfram|Alpha appliance is a great solution
for that. And what it can do is allow you to basically ask natural language
questions on your datasets, in the same way that you can use Wolfram|Alpha on
the datasets that we already have available.
So another great thing in the Wolfram Language is the computable document
format. And it allows you to very quickly and easily build dynamically
interactive interfaces. That could be valuable for a number of reasons. So
for one reason, it might help you gain visual insight into a problem or a
model or something like that. But it might also help you just quickly build
an interactive tool or an application that you can then share with your
colleagues.
And the nice thing is that they don't have to actually know how to use
Mathematica because what you're giving them as a CDF is basically just a
point-and-click application that's kind of running the language under the
hood essentially.
So CDFs are basically largely powered by the manipulate function in
Mathematica. And manipulate is a very high-level function. So this is kind
of a simple example of it. So I'll just walk through this quickly.
So of course there's a function in Mathematica called the plot function. Let
me take out F here. So if we wanted to just create kind of a trivial plot
here, we would just use this plot function like this. But if we wrapped a
manipulate around this entire thing and just added another variable which we
would like to manipulate, and then we just need to give this some range. So
let's just say 0 to 10.
So this creates this sort of dynamically interactive little tool down here.
So what you could actually do now is now take this tool and save it as a .CDF
file. And then you can send that file to basically anybody and all they need
to open and interact with that interactive tool that you've deployed is the
free CDF player that anybody can download from our Web site.
So, again, they don't need to know how to use Mathematica. They don't need
to have the software. You can just kind of freely and easily share it with
them.
So of course this is a very trivial example using manipulate. But there are
a number of other kind of examples down here that might give you a better
idea of kind of the full spectrum of tools and applications that you can
build with this manipulate function. So they can have all sorts of
interfaces. And I won't go into all of these, but you can see a couple of
these here. And I'm sure Michael will show a couple more too.
So really any notebook can be saved as a CDF file. So that means that a CDF
file does not need to only contain one of those little dynamically
interactive tools. It can also contain images and text and so on.
So basically what that means is you can also build very aesthetic reports.
And those reports could be CDF files. And, again, anybody with the free CDF
player can access those.
Let me jump ahead here. So there's actually two levels of CDF. There's the
basic CDF, free CDF as it's called, and there's also enterprise CDF. So
enterprise CDF is kind of a higher tier of CDF, and it allows you to deploy
CDFs that, for example, allow the end user to input arbitrary -- insert
arbitrary inputs or perhaps load their own datasets into the CDF application
and export their own data and just do all sorts of stuff more around
connecting to other data sources and stuff that you can't do with the basic
CDF level.
So the Wolfram Cloud is something I wanted to briefly touch on at the end
here too. The Wolfram Cloud is really -- encompasses a whole suite of
Wolfram Cloud products. So one of the those, for example, is Mathematica
Online. Another is the Wolfram Programming Cloud. But the Wolfram Cloud is
really all about taking the Wolfram Language and making it available in the
cloud and making it easy for you to deploy programs that you've written in
the Wolfram Language to the cloud in such ways that -- in such a way that
it's easy for that code now to -- like to create an API from that code or
deploy a Web form or just link that code to a Java program that you have or
some other program in some other language.
So actually now at Microsoft Research everyone has access, you would just
have to request it through me or Michael, but you can have access to
Mathematica Online, which not only would give you access to Mathematica from
any computer where you don't have Mathematica installed, but it also allows
you to do things like share what are called cloud CDFs.
And cloud CDFs are essentially like CDFs, but the computation is happening
server side as opposed to at the user's kind of front end. So one of the big
differences there is that with cloud CDF your end users don't need to have
the CDF player installed. So it's a completely Web browser-based experience.
So if you do have Mathematica Online set up ->>:
We don't.
>> Nigel Hess:
>>:
Oh, we don't have Internet?
[inaudible] I think it is.
>> Nigel Hess: Okay. Well, I won't worry about it right now. So this just
kind of shows the interface for Mathematica Online. So when you set it up
and sign into it, it kind of looks like this. It was kind of created to
really create the Mathematica experience in a Web browser. So it's kind of
trying to look as much like the local installation of Mathematica as it can
be.
So if you open a notebook in Mathematica Online, this is kind of what it
looks like in a Web browser. And of course you have full access to the
documentation center that you also have in a local install of Mathematica.
So that's very nice too, and it's just kind of shown there right on the
right-hand side.
So this just shows a couple of the different ways that you can leverage the
Wolfram Cloud to deploy Wolfram Language Code in many different ways.
So one thing you can do is basically in a line of code create an API that
allows you to call to Wolfram Language Code running in the cloud basically.
So of course there's all sorts of different applications for that. You can
also -- there's an embed code function that allows you to embed a call to
Wolfram Language Code running in the cloud in JavaScript or .NET or other
languages as well.
So there's basically all sorts of different ways of deploying your Wolfram
Language Code that were not available a year or two ago but now are kind of
trivially easy to set up and get working.
So this is kind of reiterating some of the points I just went through. And
this as well. I won't spend too much time on this. So as I mentioned,
everybody at Microsoft Research has access to Mathematica Online as well. So
if you wanted to play around with deploying APIs or creating cloud CDFs that
you could like share with your manager and stuff, just let Michael or me
know, and we can help you get set up with an account for that.
So basically all we would name is your name and your e-mail address, and then
we could put in a request to get you set up with access to Mathematica
Online.
So without further ado, I will have Michael come up and get into some of the
application areas.
>> Michael Kelly: So when you go into Mathematica, what comes up first is
our Web site showing all of the various products that we've got as well as
details and information about our language and solutions to problems and the
support and learning and everything else.
So my name is Dr. Michael Kelly. I'm a technology consultant at Wolfram
Research. And what I'm primarily concerned with at Wolfram Research is their
finance platform, but I also work in other areas.
And so the general idea here is to show you the program itself. It is a very
large program indeed. So if we go and look at the help file for it, as a
starting point, this is what you see.
So each of these boxes here are actually containers. And when I click on any
of them, it then shows up various subdivisions which cover various areas.
So, for instance, if I went to look at function visualization, there would be
a guide page there and various learning resources I could go into and look at
in more detail. So I could go in, look at the options for various graphics.
And it would explain these.
And then all of the functions that are in this particular area, and I can
click on these and look at the syntax for the language.
The reason I'm showing this is that there's a uniform syntax for our
language. Everything starts with a capital. And if there's more than one
word in there, concatenate it all together.
And there are sort of common aspects such as, for instance, if we've got
functions that do 3D, we call it 3D. Whatever the name might be, it is the
most commonly used name that you would find in that area of knowledge;
namely, in mathematics and physics and finance and economics. We use all of
the same terms.
And the -- in order to make everything easier, everything has the same
format. So this is particularly important because Mathematica uses pattern
matching. It's the only language that does so. It goes into the very
structure of the language to try to work out basically what is the input that
you're feeding it into.
And it breaks it down into these components and evaluates each of them in
turn, unless of course you're using parallel programming, which is also built
into our language, in which case it will stream across each of the components
and calculate them individually on a different kernel.
And as Nigel was saying, we have very beautiful pictures. And of course 3D
pictures, we can move these around in real time in the 3D space.
So to go back again, and what I'll just do is move through some of the
functionality. There's over 5,000 functions in Mathematica. Then we have
add-ons, which add on about another 5,000 functions. And then in addition to
that, Mathematica talks to our sister program called Wolfram|Alpha. And it
is an attempt to basically make all of human knowledge accessible within our
language.
So everything that will be shown here is, in fact, a component of
Mathematica, the entire notebook, all of the visualizations. So here, for
instance, what I'm doing is sort of zooming into a 3D network and then as we
move along, it basically -- the network unfolds.
All of these functions, you know, all of this capability are particular
functions in Mathematica. That one was the manipulate function, which Nigel
was just showing previously.
So a number of companies use our product in research and development. We're
also used quite a lot in finance by all these banks and hedge funds. And, as
Nigel was saying, it's basically one integrated system all built around the
Wolfram Language in Mathematica, whether it connects to system interfaces,
whether they be on the Web or in the cloud, the visualization in graphics,
the dynamic interactivity that includes manipulate tab view functions, our
core language underneath everything.
Behind it all of course is that Mathematica is symbolic as well as numeric.
That means that you can actually take mathematical equations almost precisely
as you see them in a mass text and then put them into our code. And, in
fact, our language itself is used by the World Wide Web. It's what Wikipedia
uses in its math portal.
So we cover a wide range of areas -- optimization, control systems. So
compared to many other programs, what we have is that everything is built
into Mathematica. They call upon core functions, which is basic to modern
mathematics. And it is these that form the basis for everything that's done.
And it can be anything -- wavelet analysis, graphics, finance, GPU
programming, the lot.
So as for the general outline, Nigel's already explained this, but we have
Mathematica, gridMathematica, which is an extension. As I said, Mathematica
is a parallel programming language. And so it can be extended not only to
additional kernels on your computer but to additional kernels which are
external. So to a grid, to a cluster.
And it can also be heterogeneous. In other words, you can have CPUs as well
as GPUs, both CUDA and OpenCL. We have it on the Web. We have a finance
platform as an extension of Mathematica and has additional financial
capabilities. We have a system modeler. So this is -- somewhat corresponds
to MATLAB's Simulink program.
Our results can be distributed for free using a CDF player. This is more or
less our -- it has commonalities with Adobe's free reader except ours is
interactive and does actual calculations for you. It's not static. And of
course for -- if you've got very large projects and you want to debug your
programs, we've got the workbench. But we also have debugging built in at a
fundamental level in the program itself.
So as you can see here, the program has rapidly developed over the last few
years. And, in fact, the last edition has brought in more functions than
there were in the original edition.
So it can cover a wide range of features and including these marvelous
graphics. So as I click on -- and, again, to remind you, everything here is
actually a Mathematica option function. Everything is really being embedded
in a series of what are called tab views. So here's an example of line
fitting. Here we've got metrics algebra, calculations. So this basically -this is what's called a sparse array, and we have sparse arrays built into
Mathematica. And you can do calculations over these.
In this instance, we've got 3D histograms where the height corresponds to the
density, the number of elements in each separate area. We've got
interpolation, both 2D -- well, you know, N dimensional. We can study
bifurcation because we've got discrete mathematics as well as continuous
mathematics. We've got wavelet analysis that allows us to do wavelet
transforms.
We've got vector fields. So you can actually have 2D and 3D analysis of
vector fields. And all of the corresponding functions such as the grid, div,
grad, and curl. We can solve partial differential equations and how they
evolve through time and plot them up.
Okay. We've got control systems and important capabilities such as locus
plotting. So this is a problem that basically we've got -- what's happening
here is that we've got a density plot, more or less a cut through a
hyperdimensional space, and the dots here are the various points that in an
optimization program as it's going, looking for either the maxima or the
minima.
And we've got image processing and also graph theory and network analysis.
So these are just some of the areas that are covered in Mathematica.
So to start off, we've got the -- as I said, Mathematica is symbolic as well
as numeric. So the only other leading major symbolic program is Maple. But
it's nowhere as extensive as Mathematica.
And so Mathematica has built in a vast amount of mathematical knowledge. So
things like symbolic sums here. So I'm taking the sum of the Kth powers, so
the first N Kth powers. And notice there are no actual numbers in here. And
as I move the cursor along here, you will see that it will do the actual
calculations for different values. So this is the sum of the fourth powers
for instance and so on. Third powers. The seventh powers.
We have symbolic limits. So here we're approaching the limit of 1 on X both
from the left and the right. And you see gives different values.
Power series expansions. So here the [inaudible] expansion just gives us the
tangent at 0. Now as I move this along, okay, you would get better and
better fields at every point along the curve.
We can solve differential equations. So here I've got an entirely arbitrary
cubic equation. And starting at values for A, B, C, and D, the coefficients
here all being minus five, I get a simple solution. Now as I change this, it
will solve for any possible of the values.
cylindrical decomposition.
And it does this using
We have wonderful 3D graphics. But I can also do the partial derivatives.
So here's the derivative with respect to the X axis and with respect to the Y
axis. And there's a number of textbooks now teaching mathematics that use
this capability.
We can do exact integration. So this is symbolic integration. So we've got
cause raised, cause of X raised to the power M, and sign of delta plus X to
the power N. And as I change the values for M and N, you can see the
integral changes.
This is unmatched by anywhere.
as the standard.
Now all the textbooks basically refer to us
And I don't know what happened there, but we'll go back.
So in addition to that, Mathematica also has been very famous in solving a
large number of computational problems. So this was a challenge given by
SIAM. And the problem here of course is that we've got infinite repetitions
of the function as we get close to 0. So as you can see here, when X goes to
0, this is unmeasurable, and so is the coefficient.
But Mathematica is capable of calculating this. This is the plot. And as
you can see here, here's the actual integral. And this is the time that it
took to do it. So it's a very fast calculation.
We can do other things. So in addition to -- because Mathematica is symbolic
as well as numeric, one of the capabilities of this is that it's built upon
transformation rules and functions. And the thing that links these two is
what's call the lambda calculus. So we have a way of representing the lambda
calculus, which is commonly used in artificial intelligence. We use the
ampersand to indicate that we're using a lambda function and the hash
argument to represent any argument at all.
So this is the function that takes its argument, subtracts four fifths and
multiplies by 5. And what I'm doing here is looking at what would be all of
the values that were generated as I start applying this function to one, and
then continue doing it 4,000 times. So this -- so initially I'm just getting
the values of 1 and minus 1 cycling around.
But what we can also do is because Mathematica is symbolic, we can start with
things like pi, the transcendental number. This is very soon after pi day.
And you notice that when I first call this up, Mathematica doesn't do
anything. It doesn't try to calculate it. The reason is that there's no
finite representation of it. I can get -- I can ask in particular, for
instance, to give me an approximation to 5,000 decimal places. Probably
using one of the theorems of Ramanujan.
But what I could also do is say I'd like it to K decimal places. And then
what I'd like to do is allow K to vary as K goes from 10 to 10,000 in
increments of 20, for instance. And then see how it changes.
Now, as it starts off, here's 10 decimal places. And then I can move this.
And as I do so, we're now at 1,180 places, and it just goes on down the
screen here. So it can be to any degree of accuracy.
So we have both infinite precision and infinite accuracy. Of course it's
going to use up more of your computer memory and will take longer to
calculate, but for important problems and for detailed calculations, such as,
for instance, calculated the friction over an airfoil, things like this where
you need absolute accuracy, then Mathematica is unbeatable.
So this is an image that Nigel had already shown you, showing that our
notebook is just another object within Mathematica, but it allows you to mix
both code and both the input and the output and comments.
Here, of course, so dynamic interactivity is very important. So this is now
going back again, solving an equation, changing the parameters in the
differential equation. As I do so, it changes the structure of the surface.
And I can increase the solution range.
All right. All right. So this next one, it took a couple seconds because
I'm pulling in a very large dataset for analysis. So basically what I'm
concerned with here is just showing some of the functionality of our
statistical analysis.
So the simplest of course basic descriptive statistics. Here each of these
tabs represents a different dataset. And every time I click on it, it will
do all of these calculations over this large dataset. Or it should. Okay.
So what's happening here is that we're getting all of the main
characteristics -- mean, standard deviation, dimensions. We've got a basic
[inaudible] plot. We're tracking a trend here. We've got a histogram.
We've got a box-whisker plot. We've got a distribution plot. Okay. We've
also got a discrete plot. And here we've got various degrees of fitting for
both the PDF and the CDF.
We've got linear and long linear and generalized linear model fitting. So
you can feed in the data, fit it to a linear model. And then we use aspects
of object-oriented programming, define out all of the properties that might
go with this. It can be the design matrix, the residuals, the hat diagonal.
We can look at the mean prediction bands, the correlation matrix. Absolutely
everything. And then we can carry out diagnostics.
And this holds for nonlinear fitting as well as generalized linear fitting.
So here you see we've got a very large dataset, really a combination of
sinusoidal functions and we can work out what are all of the coefficients
involved here. It comes back, says this is the model. This is all of these
properties. We can ask about it, and we can do diagnostics on it.
same thing for general -- generalized linear model fitting.
And the
Hypothesis testing. Again, you can feed in data. You could say, well, let's
feed it to the normal distribution or a log logistic distribution. And as it
does so -- so in this particular case we're looking at snowfall in Buffalo,
New York. You can see here how it's calculated the parameters of that
distribution and then does a statistical test to find out how accurate this
test is.
And these are some of the distributions we've got. We've got both
univariate, multivariate, continuous, and discrete. And when I look at all
of them here, so we're looking at their CDS, I can choose one here, say like
the Gumbel distribution. Let's go down and have a look at it and then change
its parameters. And as I do so, it will update the structure.
We also have random processes. So the way in which we think about random
processes is that if you allow these to evaluate through time or space and
then I take a slice, slice distribution through it, then I'll be identifying
how that distribution behaves at any particular moment in time. So it's an
extension of the concept of distributions through space and time.
And this particular thing I'm looking at, the stochastic probabilities
associated with -- so as I change the property of A winning, it's going to
change the structure of the game, depending upon where you start.
So to give you some examples here. So here I've got some data. It's a list
of pairs. This is the X value, this is the input. And the Y value, the
output. I feed that in. Then I construct a simple model. These
coefficients here are going to be fit into a second order partial
differential equation that will also satisfy the initial conditions. I will
solve this for the function, the dependent variable, for a range of values of
X. And then I will substitute this value in to find out what is the solution
to the dependent variable Y.
I return this as a model that depends upon the inputs A, B, and C. So this
just defines a very general model depending upon the inputs. And then I'm
saying, okay, let's apply this model and let's take one of its values, X, and
let's fit this model now, a function, so this is a function of X, apply it to
the data, and try to solve for the inputs here.
I better just make sure that there's no -- from any previous work that sees
also a free variable. And then try to solve this model. And it comes back
here with a solution. It also tells me that perhaps I should increase my
accuracy goal. And but, anyway, as you can see here, the solution is quite
accurate. It's a very general model the data was just arbitrarily chosen.
This picture here really is quite out dated now, but it shows how we compared
with other very well-known statistical -- let me make it a bit bigger. So
basically this is just a list of some of the distributions.
to which they're covered by other programs.
And the extent
So of course the most famous other program for statistics is R. And as you
can see, this is the light blue and this is this region here. So it cover
less than a quarter. And that was two versions back of Mathematica. So we
cover more distributions than can be found in R.
So basically how
particular case,
algae blooms and
this I just say,
file. I load in
can all of this be used to do an analysis? So in this
what I'm doing is importing a very large data file about
then looking at the chemical components within it. So in
look, this is the headings in the first row of our data
the data, and making sure to remove any missing elements.
Then I can set up a grid. This grid will give me basically an analysis of
all of the elements. It's actually sort of squashed at the moment to fit
into this format for the projector, but it has a breakdown. So this is the
data as we can see it, the season, the species, and all of the various
chemical elements that are involved.
But I can do an analysis of this, and it's so simple. All I now do is map
onto the raw data, say let the columns be as specified above. And now use in
a function that allows me to build up a dataset by threading, by basically
using the concept of association from Hadoop to build associations between
the names of each column and the values that they take and now stick it
directly into a dataset.
And now we get these summary statistics. So I can look at each individual
season and find how many elements, the size, the speed of growth of the algae
blooms. I can look at all of the -- so now I'm doing model fitting for each
of the chemical compounds, ammonia, so on, here, finding their median values.
And these are the other variables in our model. Basically this is what
the -- I can go through now, look at each of the various things here. So
here's hydrogen. Here's the number of bins I'd like to take. And how I'm
going to fit it, whether with a normal distribution or a Weibull
distribution, for instance, and see if I get a better fit to my model.
And then here is the correlation table. And the analysis is very
straightforward, very straightforward functions that are built into
Mathematica. As you'd expect, correlation in Mathematica is just
correlation. If I wanted to go in and find the particular function itself,
okay, here it is. And as you can see, it does it both symbolically as well
as numerically.
So we can import data using the import function. And as Nigel showed, that
we have about 180 different import/export types. I can load the data in, I
can gather it to analyze it, and then build up a bar chart of a structure of
the data in. So it's very simple to do.
We also have time series that allows for time series analysis. We can see
whether this -- so we've got a financial data function. Let's just see
how -- make sure that that actually went in. Yeah. So here's the data as
it's actually come in. Basically it is a list of pairs. Each pair consists
of the first element being a date and then its value. Okay. So this is, in
fact, the financial data for Starbucks, the closing data, going from the
beginning to the end of 2013 each week.
So what I can do is go into this data, just separate. So now I'm taking out
all of the actual values, sticking into a time series function, and looking
at all of the stock values as a time series object. And then I can plot them
up.
I can also estimate the process. So here are the stocks. And I'm saying,
well, you know, let's now use some Box-Jenkins analysis. In other words,
let's try to find the ARIMA process, the autoregressive integrated moving
average process. So we've got two factors that go into the autoregressive -first order integration and one moving average component. And let's use as
an estimator the maximum conditional likelihood.
If I don't specify a particular process estimator, Mathematica will look at
the structure of the data and then use this to work out what is the optimal
method to use. So it comes back, specifies this. Now, once I've got the
estimated process and the actual data, I can then just quite simply use one
function, time series forecast.
So starting with the data and the estimated process, but let's predict what
the next 26 values will be. And now let's plot up the stocks and the
forecasted value. And you can see this is what it's saying will happen. So
Starbucks, according to this, was going to make a lot of money in 2014.
We've got a whole range of functions to do with graph theory. And in this
particular one, we can use functions, for instance, such as hit centrality
which looks at the list of authority in hub centralities in any graph. And
in doing so, I can go into the Web site, take these, feed in the data from
the Web site, and do an analysis about how the information is flowing through
that system.
We also have image processing and machine learning built into Mathematica.
So in this particular case here, you can see we've got a number of images.
And what I've done is taken each image and associated it with a tag. I'm
trying to classify whether it's going to be -- it's a picture taken during
the night or during the day.
And I just feed this now directly into our classify function. And what it
will do is return a classifier function that used logistic regression and
identified that there were two classes, night and day, and then I can feed in
a bunch of images and say let's build a grid of images of this and find out
basically, you know, which one is going to be night, which one is going to be
day and see how this goes. And here it is. So it breaks it down and quite
correctly determines that these -- so these are images that were not part of
its input, but it has taught itself basically to distinguish in pictures
between night and day.
We've also got image segmentation. So here I can start with an image. I can
provide -- so I could say let's -- so let's do this and now let's remove the
background in our image. Okay. And then I can compose an image. What I'm
doing here is saying basically let's go in and look at various cloud types.
And let's now stick these, compose this with the foreground that we did and
rebuild the previous image. This is taking a second, just as it went. And
so here is the new image.
So additional functionality include enhanced color support for 3D image, as
well as segmentation, feature detection, and so on.
So one of the new sort of image processing functions, we can start with an
image, okay, I can do -- break it down to look at all of the various
components and then color these in order to analyze this in greater depth.
So basically getting component measurements.
So in this particular case, for instance, it could be we're trying to
determine something like sickle cell anemia by looking at cells in a blood
stain. And now we count the number for each of the cells and look at their
elongation. I can now calculate this directly in Mathematica by saying,
okay, let's draw a histogram where basically let's just count the number of
elements that are of a certain length. And, in fact, let's go into it a bit
more detail and ask how many cells -- let's select only those cells that are
of elongation greater than 0.5, which cells satisfy that criteria.
More machine learning. There's a number of different methods. So here what
we're doing is we've got digit classification. So basically this is a
handwriting version of the first ten digits. We use this to train the
function and then come back and apply it to this list here. And you'll
notice that it's got everything right except for 6, which it interpreted it
as a 4.
So we go and ask it. Tell me what are the top probabilities associated with
6. If you didn't think it was 6, why did you think it was 4? And it comes
back and says, well, I thought it was 4 because it has a great -- I also
thought there was a certain probability of it being 6 and 0, and these are
their associated probabilities. So I assumed it was 4 because it had a
higher probability associated with it.
So we can analyze data. So here's a random generation of data from
multinomial distribution. And then I will do a cluster analysis of this
based upon various versions of the classification analysis with the logistic
regression, nearest neighbors, random forest, and the rest.
Now we also, as I was saying beforehand, we have a link to Hadoop. But in
version 10 -- this was introduced in version 9. In version 10 we don't
introduce the lot of the functions that are in Hadoop, the concept of
association we've already shown, but we can also deal with HDFS files.
have the MapReduce function built in now into Mathematica.
We
And you can apply this to things like Jane Austen's Pride and Prejudice to
give a frequency distribution of all of the 4-grams in that novel.
I think Nigel has already shown this. We cover a wide range of functional
visualization, 2D, 3D, charting and statistical. And this includes things
like 2D plots, parametric plot with two variables, contour plot, a density
plot, a region plot. This involves new concepts that have been built into
Mathematica that allows it to study statistical aspects of measure such as
Lebesgue measure.
And then we've got polar plots, vector plots, stream plots, and all of their
3D counterparts. And charting. Okay. Both 2D and 3D, including bubble -3D bubble charts and statistical visualization, whether their histograms, 3D
histograms, paired histograms. And also you can smooth your histograms with
a kernel of your own choice. It doesn't have to be a Gaussian kernel. You
can define any such function or a matrix.
The visualization functions in Mathematica are very powerful. So here we've
got two functions, half a sine and twice a cause function. Okay. Show that
this is actually happening in real time. This is the same thing calculated
over a wider range.
We can do functions parametrically. So this is the X and the Y component
expressed as functions of the one parameter T. Okay. We can do things
like -- now, what I can do also is take functions -- these are sinusoidal
functions. They depend upon one parameter, theta, as it goes from 0 to 2 pi
radians.
But what I'm going to do is associate with each image a tool tip. So now as
I pass over this, okay, it will tell me what the actual function is. Or it
should do. Ah, there it is.
Okay. We can do parametric plots in 3D. So here are the X, Y, and Z values
that depend now upon two parameters. As these vary, we get this cone. Okay.
Here we've got again a contour plot in 3D. Depends upon three variables.
And I can also change things like the color style and everything like this.
And also, once I produce it, now being an object in the 3D space, I can move
it around.
And we've got other metrics plots, regions plots, vector plots, density
plots. And in 3D, revolution plots, parametric plots, contour plots, and so
on. Basically every possibly type of function and data visualization is
built into Mathematica.
For histograms, so we can feed in -- so here's a normal distribution. Here
is its mean and standard deviation. I can generate any random real values
from this. So 200 of them. And then stick them into a histogram.
So as you can see, one line of code does a hell of a lot. And this is
typical of Mathematica. That you don't have to actually think about writing
up the description of the normal distribution. That's a built-in function.
You merely have to specify it. If you then want to do Monte Carlo estimation
of it, you just say I'd like random real values, specify the number, and then
say I'd like to see this stuck into a histogram. And then you can even
specify any aspects or properties of that histogram.
Here, for instance, I've got overlays. I'm going to do Poisson
distributions. So I'm basically drawing the histogram of the Poisson
distribution. And then I'm asking the program, well, since it already knows
Poisson distribution symbolically, then tell me what their PDF is, their
probability density function.
Now, let I vary from 1 up to 5. So I'm going to have five such distributions
and I'm going to draw their probability density plot against the histogram
plot. As you can see, we regenerate all of this. It shows that the Monte
Carlo estimation that's being used by Mathematica is quite accurate.
So this is more or less like a catalog of all our different functions.
Plotting curves in the plane, representation of surfaces and so on. I can
click on any of these and actually go to this area in our help file. And
always these are all interactive as well. It will recalculate it.
And for every function, it always has options. So the notion here okay is
that there is a general function, but of course you might want to modify it
in any way. You can go down more or less into the guts of the function
itself and have it modify its behavior according to different methods. You
can even write your own methods as special functions.
But particularly for graphics functions, you should be able to modify them in
a way that indicates how it is you want to do so. So here, for instance,
we're just doing -- we're plotting sine and cosine functions, and I'd like to
fill it. So the automatic filling would be to the X axis or I could fill it
to the top as the maximum value of the two functions, or I could fill it to a
particular point. Or here I could say let's fill in the space between the
two curves and let's do so in an oscillating manner from yellow to green.
So we have chart element functions here. So I've got different charts, pie
charts, and I can then label them. You can label charts in many different
ways by basically saying the actual position in the 2D or 3D space where you
want to put the positions of the markers.
And we can also generate reports from this. So here's a report, for
instance, where we've got the G8. We're looking at the populations of the G8
countries. And we color their positions on the map and then build histograms
out of their national flags.
So we can do things -- this is how easy it is to build up functions in
Mathematica. I want to compare a distribution with its histogram. So I
build a compare histogram 3D function. This is a couple lines of code. I
then build in or call upon our multinormal distribution. And now let's
compare this multinormal distribution with the PDF. And here we have -- so
the PDF of the multinormal distribution -- let me make this a little
bigger -- and its histogram.
And you can see how the histogram more or less encapsulates the actual PDF
function. And we can do the same thing for the CDF. Again, let's make this
bigger. And again you can see that the histograms encapsulate the actual CDF
function, acting like a stepping stones over it.
We can also take textures and include these as overlays on the surface of the
map. And these functionalities can also be kept and stored and sent off to
create real objects such as plastic coloring and stuff like that.
Now, we have financial capabilities. So here I've got an interactive chart.
That allows me to load in values from Amazon. I can say I'd like to look at
the volume; the RSI, the relative strength index; and the Bollinger Bands.
I can choose other types of indicators as well. I could say let's go and go
down here and let's look at momentum type indicators and let's add a moving
average convergence/divergence. And here you can see it. Here's the MACD.
And as I move my cursor over the graph, you can see the corresponding points
on all of the financial indicators and the actual graph of values.
And we can choose which region I'd like to look at within the history of the
stock.
We've got geospatial visualizations. So this is actually being used online,
I think, by a company to look at Medicare beneficiaries with heart disease
around the country. And also the same for diabetes and the rest. And in
that particular case, okay, when I went and clicked on the case for diabetes,
then I think it's now going offline and looking at how this was generated.
So here is -- yes. And here is the site online.
within Mathematica.
All of this work was done
Okay. We can drill down into it. We can build interactive maps. So here,
for instance, I've put in let's go into Chicago. I want satellite images. I
want to zoom out, zoom in. Okay. Perhaps I want a roadmap instead. Or I'd
like to actually look at the terrain itself or some type of hybrid function.
So basically we could do all of the same type of stuff that you already find
in Google Maps.
You can create your own interactive interfaces. So basically these are
examples that all use the manipulate function. So the manipulate function is
worth looking at in its own right. And so I'll just quickly go into the help
file to look at it.
Okay. This is basically a wrapper function. So it's like a generalized Java
GUI that allows you to feed in any Mathematica code that describes a model
with three parameters. So we've got here a situation where we're plotting a
sine function, and we've got the frequency A, and I'm allowing it to change.
If I just allowed it to change up to 4, for instance, then I'd get a
different function here. And now as I move the frequency here, you can see
the function is changing.
So this is a very powerful function. What it does is allow you to put in a
new model and see how the choice of the parameters effect your result. So
here I want to map for instance the cosine function. I can choose the
various coloring that I want, the filling that I want, and so on and so
forth.
I can study a circular, rotary, or star engine. I can change the number of
cylinders. Let's go for seven such cylinders. And then I can have it cycle
through various movements or I could say okay, let's just start playing it.
This was a problem that was shown beforehand. What we're doing here is
solving a partial differential equation interest the electrostatic potential
between two point charges, Q1 and Q2, as I change their strengths, their
relative strengths, so the electric field changes.
And this is a predator-prey model, where we've got foxes feeding on rabbits.
I can change the growth rate for the rabbits and their mobility so they can
get out of the range of the foxes. And the same thing for the foxes. And
the number of steps that we want to go through.
And then I can run the simulation. I can stop it, reset it and run it
So all of these -- so in this particular case, the predator-prey model
linked second-order PDE, and I'm just going through all of the various
in it. So the output for one stage feeds into the input of the second
again.
is a
stages
stage.
The other thing that Mathematica is capable of doing, of course, is talking
to our online program, Wolfram|Alpha. So that now all of the capabilities of
Wolfram|Alpha can be accessed programmatically within Mathematica.
So the way this is done now is to define various entities. So here what I'm
doing is looking at the entity which is of the type or class administrative
division. And I want the entity list, the list of all such entities that are
basically in France. And I also want the corresponding list of
administrative divisions that are in Illinois in the United States.
When I evaluate each of these, you can see all of the various regions in
France. And as I pass my cursor over them, you can see that they are both --
they are all entity objects of the type that I requested.
for in Illinois.
And the same thing
Now, we can use this new function in Mathematica, the GO region plot and also
its value plot. So I'm now going to go into France, look at all of the
departments that we specified. And what I'm going to do is associate with
each of these some just basically a random number. So what I'm trying to do
is identify each of the different administrative divisions within France,
giving them different random numbers, and then turning this into a value plot
that's going to be based upon a color scheme.
Or what I could do is associate it instead of with a random number with the
actual population size. You can see that largely population of France is
uniform except, say, in the south, in the southwest region and in the
northwest regions.
I can go in here and do a similar type thing saying, all right, let's
associate each of these countries with different numbers, and then these
would actually be the sort of heat colorations for each of the countries, and
then I can put all of these into a geo plot in order to color them. And the
same thing can be done again with all the administrative divisions for
Illinois and Vermont.
Or I could associate -- I could say, okay, let's find all of the different
countries within Africa, all the different nation states in Africa. And then
I'm going to associate with each of these -- I'm going to get each common
name, form a polygon out of it, and then plot it all up.
And the same thing can be done. So again, all of these -- these are
basically if I was to go into Wolfram|Alpha, and Wolfram|Alpha allows us to
ask any type of question. So we could, say, for instance, go in and say,
look, tell me about Mona Lisa. Okay. So we enter that value in. And it
comes back and it says, well, it's an artwork. Do you agree? Yes, I say
yes, I accept this.
And as you can see here, as I leave the cursor over, it says it's an entity
type object. Now, being an entity, it has properties. So I can ask it about
one of its properties, or any of its properties, and say -- that's got to be
a capital -- tell me about it. And it says okay, yes. So ->>:
Five minutes, maybe.
>> Michael Kelly: Five minutes. Okay. So I can get images, things like
this about it. I should be able to say -- let's copy that, put it here. And
then say yes, tell me the -- should be an image. Yes. So let's do just
image there. Okay. And there's the image for the Mona Lisa.
So in this way I can mix and match, as it were, real-world knowledge with
mathematical capabilities. No, I don't want to go into that.
We've also got integration with R. So we can do patent classification, for
instance, in R, we can load in R. We've got a link to it. This is a two-way
link. You can do your coding within Mathematica and then have it calculate
these things directly using R functions as well. So here we've got singular
value decomposition looking for the two principal components in analysis of
data. Here I call in various other external libraries and then do a similar
type of classification analysis within R.
As I said, we've also got the Hadoop package, and we can load that in. We've
got similar corresponding functions in Mathematica to things that we've got
in Hadoop. So, for instance, we've got the function -- the MapReduce
function in Mathematica. It's called group by. So suppose I had a list of
functions and what I would like to do is associate each element with the way
it's -- its result under the action of the function F. So F is unspecified
at this time. So it sets up an association between the application of the
function F and those elements that are going to be the same as it.
So if I applied, for instance, to this range of negative and positive numbers
the absolute value, you'd find that the absolute value 3 is associated with
both minus 3 and plus 3. The absolute value 2 is associated with both minus
2 and plus 2. I could go into a list of pairs, and I'd see that for the
original list of pairs, both the pair AB and AC have the same common element
A. And so on. Or I could look at the elements that have the same common
last element, C.
So all of these type of functions which occur in Hadoop also can be applied
in Mathematica. I can merge common elements together. So here, for
instance, I've got the same key value, A, being associated with X and Y. Now
when I merge them under the application, the function F, it would be that A
is going to be associated with both F applied to X and Y, whereas B stands
alone.
So not only can you merge elements that share common features but you can
apply a function to it and then merge those features that are the same under
the action of some transformation function.
So here, for instance, you notice we've got the same two keys, A and B, but
associated with different values. I'd like to merge both datasets in such a
way that what I'm now interested in is the total of the common values. So we
should get A associated with the sum of these two elements, 1 and 5, that is
6, and B associated with the sum of 2 and 10, 12.
And we've got application development. So basically using the capabilities
that I've already outlined, the use of the manipulate function. We can do -load in the data. So this is an example of an actual model that was built
for a consult. We can study things like the oil spill in the Gulf of Mexico.
This entire thing is done in Mathematica. In fact, as I click on it here, I
bring up the notebook and I can run it, run through it and study this in
detail.
So in the limited time, I really haven't got all that much -- I can't cover
all of the capabilities in Mathematica. Let me just end with just one
further comment, which is really ->> Nigel Hess: Maybe put our e-mails up on there? Because I didn't get a
chance to do that just in case they have questions or -- or just my e-mail.
>> Michael Kelly: I've just got one thing. I thought we'd just actually
show the cloud deployment. So one of the new aspects that Mathematica has is
deployment to the cloud. So what you can do is do an evaluation function
within Mathematica and send the result to the cloud. So I could, for
instance -- let's go in, get a file that has a piece of code in it, very
simple code. I will bring this out, make it a bit bigger.
And so what I'm going to do here is, yes -- so basically I'm going to create
a piece of code. What this code is going to do is say let's create a form in
the cloud where you can specify the stock and the number of days of its
history that you want to analyze. It will then go and look up the financial
data for that stock and the number of days back in history. And it will put
up a title to this, create a gift function, and put it in a separate -- give
it a separate directory.
Now, this is then being created. I can click on this. It comes to the
cloud, and I could say yes, tell me about Microsoft and let's look at its
value, say, over the last 360 days. And let's submit that. And here we are.
So here is the stock price of Microsoft over the last 360 days.
So going to the very end, this is your thing at the very end, Nigel?
>> Nigel Hess: It might be on the first slide. Or we could just -- you
could just type it in there, nigelh@wolfram.com.
>> Michael Kelly: So -- oh, no, it doesn't seem to want to let me do it.
But, anyway, let's do it here. So for Nigel, if you need to ask any further
questions, his e-mail is nigelh@wolfram.com. And for myself, it's
mkelly@wolfram.com. Nigel's concerned with sales; I'm more concerned with
the -- and let me make this a bit bigger. And I'm concerned mainly with
consulting and also the solution of financial and statistical problems.
But thank you very much for having us here and boring you to death. And I
hope that if you have any further questions or topics of interest, as I said,
Mathematica is a very, very large program indeed. But because of it, it
really has such a wide range of functions that cover every possible area of
human knowledge and scientific application.
And many of the world's leading firms, both intellectual, financial,
engineering, and otherwise, use our products because they make it so much
easier to solve problems than going down and using procedural programs at a
basic level. It takes time. You have to find the libraries. You have to
find the people. Here everything is already precalculated, thought out in
detail and interlocking and interlinking.
So thank you very much.