Installation of EMI, GAMS
and Python
Information paper
23 November 2012
Market Performance
Installation of EMI, GAMS and Python - Nov 2012
Installation guidelines
Contents
1
Introduction
Minimum system requirements
1
1
2
Microsoft .NET Framework
1
3
Installation of GAMS
Put the GAMS system directory on the path
2
4
4
Installation of EMI
6
5
Python
Basic Python installation
Installing Python packages
Installing the py-gdx utilties
Rebuilding the Python bindings
6
7
7
8
10
Glossary of abbreviations and terms
13
Figures
Figure 1
Create a GAMS project file
3
Figure 2
Run the model called trnsport
4
Figure 3
Objective function value for the model called trnsport
4
Figure 4
Adding the GAMS system directory to the path
5
Figure 5
Windows, GAMS and Python versions
6
Figure 6
Installing a package with easy_install
7
Figure 7
Getting help with gdx_extract_csv.py
9
Figure 8
Extracting data from a GDX file with gdx_extract_csv.py
9
Figure 9
Listing the symbols in a GDX file with gdx_list_symbols.py
10
Figure 10 Failed attempt to invoke gdxsetup.py
11
Figure 11 Successful attempt to invoke gdxsetup.py
11
Installation of EMI, GAMS and Python - Nov 2012
ii
23 November 2012 8.22 a.m.
Installation guidelines
1
Introduction
1.1
This document provides information to assist with the installation of EMI, GAMS and Python on a
computer running Microsoft Windows. It is advisable to read this entire document before
proceeding to install EMI.
1.2
The Electricity Market Information (EMI) software is a Windows-based application developed by
the Electricity Authority (Authority). Its purpose is to make the Authority's models easy to use and
accessible beyond the GAMS programming community. Most of the models accessible through
EMI also require the installation of the General Algebraic Modelling System (GAMS) software. A
GAMS license is also required.
1.3
Additional information about EMI can be found at www.reports.ea.govt.nz. The GAMS software
and information regarding a GAMS license can be found at www.gams.com.
1.4
Installation of Python is optional; neither EMI nor GAMS require the use of Python. However,
advanced users of the Authority’s models will probably want to substantially manipulate input
data, edit GDX files, or visualise model outputs. Familiarity with a general-purpose, interpreted
high-level programming language such as Python would therefore be very useful. Additional
information about Python can be found at http://python.org/.
1.5
Finally, advanced users of the Authority’s models will find it helpful to install a suitable text editor.
Minimum system requirements
1.6
Users of EMI require at least the following:
(a)
a computer running Windows XP service pack 3 or later;
(b)
a licensed copy of GAMS, version 23.43 or later;
(c)
administrator rights to install software on the computer;
(d)
read and write access to all, or at least a designated portion, of the local hard disk drive;
and
(e)
Microsoft .NET 3.5 or later.
1.7
The more memory available on the computer the better for solving the Authority’s models.
However, any modern computer will likely have at least two to four gigabytes of memory, which
will be sufficient.
1.8
It is also advisable to locate the EMI model folders on the local hard disk rather than on a shared
network drive. When the models are being solved, a significant amount of reading from and
writing to the hard disk may occur. It may be better for other network users if this activity is not
undertaken across the local network.
2
Microsoft .NET Framework
2.1
In order to use EMI, it is necessary to have the Microsoft .NET framework installed. It is advisable
to install the .NET framework before installing EMI.
2.2
If Windows 7 or later is in use, Microsoft .NET 3.5 or later will almost certainly already be installed
as part of the Windows operating system. For earlier versions of Windows, or cases where
Microsoft .NET 3.5 is not detected, the EMI installer will attempt to locate and install the
Microsoft .NET framework as part of the EMI installation process. Please do not try to prevent this
Installation of EMI, GAMS and Python - Nov 2012
1 of 13
23 November 2012 8.22 a.m.
Installation guidelines
from happening. If the installation is allowed to be completed without a successful installation of
the .NET framework having taken place, the EMI will software will not work.
3
Installation of GAMS
3.1
It is advisable to install GAMS and test that it is operating properly before installing EMI.
3.2
Obtain the appropriate GAMS installer for your system from http://www.gams.com/download/.
Windows installation notes are also available from the GAMS website, although the installation
process is simple enough that they probably aren’t required.
3.3
Run the installer, i.e. double-click the downloaded windows_x86_xx.exe file, where xx
denotes 32- or 64-bit. When prompted for installation mode, choose the default mode.
Importantly, when prompted for the name of a directory or folder in which to install GAMS, choose
1
the c:\ root directory. For example, c:\GAMS23.9\ if you are installing GAMS version 23.9. Do
not let the installer place the GAMS system into the c:\Program files\ or the C:\Program
Files (x86)\ directories.
3.4
The location where GAMS is installed is referred to as the GAMS directory or the GAMS system
directory. This is a key concept that must be understood. When it comes to actually doing some
work using GAMS, the location where you store your GAMS model codes (or programs) and
related data files will be known as the GAMS working directory, or perhaps the current directory.
The GAMS system directory and working directory are never to be the same place. As a general
rule, stay out of the GAMS system directory and don’t use it as a place to store work files that you
create or edit.
3.5
Towards the end of the installation process, you will be asked if you have a GAMS license file. If
you do, it will probably be a text file called gamslice.txt. When prompted, instruct the installer
where to find your GAMS license file and it will be copied to your GAMS system directory. Do not
attempt to edit the GAMS license file. Note that it is possible to install GAMS without first
obtaining a GAMS license file. A license file can be placed in the GAMS system directory at any
later date.
3.6
When the installation is complete, test that your GAMS system works properly by solving one of
the models in the GAMS model library. But first, open the GAMS IDE and create a project file –
do this by selecting File | Project | New Project as indicated in Figure 1. Then navigate
to some temporary location, e.g. c:\temp and create a project file – give it any name you want.
After EMI has been installed, you can come back to this step and create a new GAMS project file
in one or all of your EMI model folders.
3.7
The GAMS project file serves a couple of purposes:
1
(a)
Its location indicates to GAMS the whereabouts of the default working directory – this is
where GAMS will store any files created as part of your GAMS job and it’s where GAMS
will look first to find the input files it needs.
(b)
The project file itself will store any customisations and options you select from within the
GAMS IDE. For this reason, it is advisable not to delete the GAMS project file, although it is
not fatal if you do.
Throughout this document we use the terms folder and directory interchangeably.
Installation of EMI, GAMS and Python - Nov 2012
2 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Figure 1
Create a GAMS project file
Source:
Electricity Authority
3.8
The GAMS project file will have the suffix .gpr. It is sensible to create a GAMS project file for
each GAMS project you work on. For example, if you use both vSPD and GEM, you might have
two GAMS project files – one each for vSPD and GEM. Hopefully it is clear by now that these two
GAMS project files will be in different locations and that only one of them will denote the active
working directory at any point in time.
3.9
It is now time to run a test model. Select File | Model Library | Open GAMS Model
Library and search for the model called trnsport (there is meant to be no ‘a’) as indicated in
Figure 2.
3.10
Selecting the model called trnsport will load it into the GAMS IDE. Hit F9 or select File |
Run. The model should solve to give an objective function value of 153.675000 – see Figure 3.
3.11
If you don’t see something similar to the image in Figure 3, where the objective function value is
clearly indicated to be 153.675000, then the installation process has probably gone awry.
3.12
Finally, at some point you may be prompted to select default solvers – do this by selecting
File | Options and choose the Solvers tab. Each model type (the columns) can be
assigned a default solver (the rows). Choose a solver with a full license, if one exists, ahead of a
demonstration (demo) license. The model types that matter for the models available in EMI at this
time are: LP, MIP, and RMIP. The cells with black dots represent the applicable solvers.
Installation of EMI, GAMS and Python - Nov 2012
3 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Figure 2
Run the model called trnsport
Source:
Electricity Authority
Figure 3
Objective function value for the model called trnsport
Source:
Electricity Authority
Put the GAMS system directory on the path
3.13
Once GAMS has been installed, it is important to make sure that the GAMS system directory is
added to the system Environment Variable called the path. Failure to ensure that GAMS is on the
path is likely to mean that EMI will be unable to find where the GAMS system is located.
3.14
The precise method for finding and editing the environment variables may depend on the version
of Windows in use. One easy approach that usually works is to open the Control Panel and in the
Installation of EMI, GAMS and Python - Nov 2012
4 of 13
23 November 2012 8.22 a.m.
Installation guidelines
search field, type environment. Then click on Edit the system environment
variables. Alternatively, the method is more or less as follows:
3.15
(a)
select Control Panel from the start menu;
(b)
choose System and security, then System, then Advanced system settings,
and then Advanced. One way or another, you need to locate and select the Advanced tab
on the System Properties dialogue box as indicated in Figure 4;
(c)
click on the Environment Variables… button;
(d)
under System variables, select Path and then click the Edit button; and then
(e)
add c:\GAMS23.9; (or whatever the file pathname to the GAMS system directory is in
your case) to the end of the string of characters, and click o.k.
Figure 4
Adding the GAMS system directory to the path
Source:
Electricity Authority
If you are unable put the GAMS system directory on the path by yourself, please seek guidance
from your IT support staff.
Installation of EMI, GAMS and Python - Nov 2012
5 of 13
23 November 2012 8.22 a.m.
Installation guidelines
4
Installation of EMI
4.1
Wait until Salvador has prepared a new EMI installer before writing this section.
4.2
What about the individual models in EMI? Do they get installed with EMI or does this happen
separately? Provide an illustration of what an EMI folder with various model folders therein will
look like, i.e. the tree structure.
5
Python
5.1
As noted above, Python is a very useful interpreted, high-level programming language. It is an
open source application so is therefore able to be freely downloaded and installed. There are
thousands of modules or packages that can be installed as part of your Python system,
depending on your specific requirements. Sadly, however, the Windows versions of Python and
Python packages do not always install as the authors claim or hope they will. Hence, it may take
a little work in order to make them operational. One reason to install Python is to make use of the
GDX utilities (py-gdx) prepared by Geoff Leyland – see https://github.com/geoffleyland/py-gdx.
5.2
The notes provided here relate to the experience of staff in the Market Performance team at the
Authority. They are provided in the hope that they may be of some help but they cannot be
guaranteed to work with every particular Windows system. For the sake of clarity, Figure 5
indicates that the computer to which these notes refer is running a 64-bit version of Windows 7, a
2
64-bit version of GAMS 23.9.1 and a 32-bit version of Python 2.7.3.
2
Figure 5
Windows, GAMS and Python versions
Source:
Electricity Authority
At the time of writing, the author was unable to get the 64-bit version of Python 2.7.3 to work properly with
the desired Python packages so reverted to a 32-bit installation.
Installation of EMI, GAMS and Python - Nov 2012
6 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Basic Python installation
5.3
Using a web browser, navigate to http://python.org/download/ and download the required installer
– the Python 2.7.3 Windows installer called python-2.7.3.msi in the present case. Run the
downloaded installer to install Python in the c:\ root directory, e.g. c:\Python27\. Following
the instructions sketched out in section 3, put the Python system directory and the Python scripts
subdirectory on the path environment variable, e.g. c:\python27 and
c:\python27\scripts.
Installing Python packages
5.4
See http://pypi.python.org/pypi for additional information about Python packages. At the time of
writing, there are over 25,000 packages available to augment the Python installation. When a
Python package is installed, the Python system directory is updated with the files necessary to
make the contents of the package available to the particular version of Python.
5.5
It is worthwhile installing a utility such as easy_install to manage the process of installing
packages. Easy_install will take care of locating and downloading the most recent version of the
package that is applicable to your particular Python system and taking care of the installation. An
example of using easy_install to add the package called statsmodels is shown in Figure 6.
Figure 6
Installing a package with easy_install
Source:
Electricity Authority
5.6
Easy_install can be downloaded from http://pypi.python.org/pypi/setuptools. For example, after
downloading and installing setuptools-0.6c11.win32-py2.7.exe, the easy_install
executable was placed in the c:\python27\scripts\ directory, whereupon it can be easily
accessed because the c:\python27\scripts\ directory is on the path.
5.7
The Market Performance team find the following packages useful: numpy, scipy, pandas,
matplotlib, pyreadline, xlrd, xlwt, and IPython. Using easy_install to load these packages as
demonstrated above yielded, respectively version 1.6.2 of numpy, 0.11.0 of scipy, 0.9.0 of
pandas, 1.2.0 of matplotlib, 2.0 of pyreadline, 0.8.0 of xlrd, 0.7.4 of xlwt and 0.13.1 of IPython.
5.8
The easy_install invocation was modified as follows when installing the IPython package:
easy_install ipython[zmq,qtconsole,notebook,test]
5.9
Just to be clear, it is not necessary to install all or, perhaps, even any of these packages to use
the py-gdx utilities. They are listed here for completeness so that there can be no confusion as to
Installation of EMI, GAMS and Python - Nov 2012
7 of 13
23 November 2012 8.22 a.m.
Installation guidelines
the state of the Python installation used by the Market Performance team. Having said that,
numpy, scipy, matplotlib, and pandas are very useful packages.
Installing the py-gdx utilties
5.10
Go to https://github.com/geoffleyland/py-gdx and click on the zip icon to download the file called
py-gdx-master.zip. Unpack it and place the Python codes (the files with the .py suffix) in the
directory where you wish to use them. Use of the py-gdx scripts is quite straightforward and is
explained on the py-gdx website. A couple of demonstrations using a GDX file called test.gdx
are presented.
5.11
The first example, shown in Figure 7, illustrates how to use the help option to obtain all of the
options available when using gdx_extract_csv.py. The essential purpose of
gdx_extract_csv.py is to extract selected symbols from a GDX file and place them in a CSV
file. The other scripts can be similarly invoked using the --help option to see an explanation of
usage and syntax.
5.12
Figure 8 illustrates how to use gdx_extract_csv.py to extract the symbol called i_fuelPrices
from the GDX file called test.gdx and write it to a CSV file called fuelPrices.csv. A typical
usage of this functionality would be to open the CSV file in Excel, edit the fuel prices, save them
back to the CSV file, and insert them back into the GDX file using gdx_insert_csv.py.
5.13
The final example, shown in Figure 9 illustrates how to use gdx_list_symbols.py to show a
list of all of the symbols (including the symbol type and domain) from the GDX file called
test.gdx.
5.14
The py-gdx utilities make use of the Python Application Programming Interface (API) distributed
by GAMS and found in the C:\GAMS23.9\apifiles directory. Because the GAMS system
directory is on the path, an invocation of any of the py-gdx utilities from any location should mean
that the GAMS Python APIs are able to be found. However, in the examples presented, that
doesn’t seem to be the case. As can be seen in Figure 8 and Figure 9, the Python invocation
requires the use of the –g c:\GAMS23.9 argument to instruct Python where to find the GAMS
system directory.
Installation of EMI, GAMS and Python - Nov 2012
8 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Figure 7
Getting help with gdx_extract_csv.py
Source:
Electricity Authority
Figure 8
Extracting data from a GDX file with gdx_extract_csv.py
Source:
Electricity Authority
Installation of EMI, GAMS and Python - Nov 2012
9 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Figure 9
Listing the symbols in a GDX file with gdx_list_symbols.py
Source:
Electricity Authority
Rebuilding the Python bindings
5.15
The Python bindings in the apifiles directory of the GAMS23.9 system directory relate to
Python 2.6 and do not work for Python 2.7. It is necessary therefore to rebuild them. Instructions
on how to do this are provided in the file called readme.txt found in
C:\GAMS23.9\apifiles. In a nutshell, the Python script called gdxsetup.py must be
invoked to re-create the C extension. But this requires a C compiler to be accessible on the
computer from where gdxsetup.py is invoked. Figure 10 illustrates what happened on the
Installation of EMI, GAMS and Python - Nov 2012
10 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Market Performance computers – the rebuild failed because a file called vcvarsall.bat,
something to do with the C compiler, could not be found.
Figure 10 Failed attempt to invoke gdxsetup.py
Source:
5.16
Electricity Authority
As it happens, Microsoft Visual Studio 2010, which has a C compiler, is present on the computer
in question so instructing Python where to find it is all that was required to successfully rebuild the
Python bindings. The command in the first line of Figure 11 is all that was required to rectify this
problem.
Figure 11 Successful attempt to invoke gdxsetup.py
Source:
Electricity Authority
5.17
As can be seen towards the end of the image in Figure 11, this process has resulted in the
appropriate modifications being made to the packages library in the Python system directory so
that GAMS and Python are able to communicate with each other via the rebuilt GAMS Python
API.
5.18
However, it may not be possible to replicate all of the above as Microsoft Visual Studio 2010 is
not freely available. But an express version is freely available from
http://www.microsoft.com/visualstudio/eng/products/visual-studio-2010-express and with a little
luck, it contains the required C compiler. Beyond this, the Authority is unable to offer any
guidance – seek help from your IT support or contact GAMS.
Installation of EMI, GAMS and Python - Nov 2012
11 of 13
23 November 2012 8.22 a.m.
Installation guidelines
5.19
Finally, it is perhaps worth noting that, as at the time of writing, the Authority is aware that a
forthcoming release of the GAMS software will contain a more developed set of Python APIs and
the issues outlined above may no longer be of any relevance.
Installation of EMI, GAMS and Python - Nov 2012
12 of 13
23 November 2012 8.22 a.m.
Installation guidelines
Glossary of abbreviations and terms
API
Application Programming Interface
Authority
Electricity Authority
CSV
Comma Separated Variable - a comma-delimited text file
EMI
The Electricity Market Information software
GAMS
General Algebraic Modelling System
GDX
GAMS Data Exchange – a GAMS-based binary file format
GEM
Generation Expansion Model
IDE
Integrated Development Environment, as in GAMS IDE
vSPD
Vectorised Scheduling, Pricing and Dispatch
Installation of EMI, GAMS and Python - Nov 2012
13 of 13
23 November 2012 8.22 a.m.