Spack HowTo

Content

Introduction

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments.

Prerequisites

Spack has very basic prerequisites:

  • Python 2.6 or higher
  • A C/C++ compiler
  • git and curl packages

Getting started

To install Spack and install your first package:

$ git clone https://github.com/llnl/spack.git
$ cd spack/bin
$ ./spack install R

Documentation

Spack provides different reference resources:

Spack in SFT

Current status

The group is currently investigating the usage of Spack within the context of the HSF project. Here is a summary of progress made so far:

  • Initial test of concept
    • Basic use of Spack
    • Limitations
    • Similarities with LCGCMake
    • Different ways to adapt Spack to our workflow
  • Installation of ROOT using Spack
    • Adding not supported packages to Spack
  • Integration with Jenkins
  • Creation of a cern-spack Spack repos
  • Scripts to install a toolchain of packages and versions.
  • Use of AFS/CVMFS compilers
  • Different builds tested in CentOS7 and SLC6 with Jenkins
  • Prepare EOS as a private Spack mirror to look for source files
  • Some package and core contributions to Spack

The rest of this document shows how to perform usual group tasks making use of this new package manager tool.

Adding new mirrors

Spack provides commands to create your own mirror. By default, Spack will download every source file from the specified url in the package file.

class R(Package):
    """R is 'GNU S', a freely available language and environment for
    statistical computing and graphics..."""

    homepage = "https://www.r-project.org"
    url = "http://cran.cnr.berkeley.edu/src/base/R-3/R-3.1.2.tar.gz"

    extendable = True

    version('3.3.1', 'f50a659738b73036e2f5635adbd229c5')
    version('3.3.0', '5a7506c8813432d1621c9725e86baf7a')
    ...
    version('3.1.2', '3af29ec06704cbd08d4ba8d69250ae74')

Sometimes, we need full control over the build process so you would rather download all these source files from your own private mirror. Creation of new mirrors can performed in many different ways. The Spack documentation shows some detailed examples:

The usual manner proposed by Spack is the following:

  • Users lists which packages will make up the mirror
  • Spack downloads the tarfile of each package from the official url (specified in the package file)
  • These tarfiles are arranged in a special directory structure so that Spack can find them.

This method can be cumbersome if you already have a mirror with tarfiles at your disposal. That is the case of the LCG build processes, where every source file is taken from a mirror in EOS. An easy solution is to rearrange the existing tarfiles with the structure that Spack expects to find.

Let's suppose the following structure:

http://myserver.com/packages/tarFiles/
  |-- Python-2.4.2.tgz   
  |-- Python-2.7.0.tgz
  |-- cmake-3.5.2.tar.gz  
  |-- mysql-5.7.11.tar.gz   
  \-- numpy-1.8.0.tar.gz     

Creating a new folder with this other structure your current software repository is ready to be used by Spack:

http://myserver.com/packages/newMirror/
  |-- Python
      |-- Python-2.4.2.tgz
      |-- Python-2.4.2.patch
      \-- Python-2.7.0.tgz
  |-- cmake
      \-- cmake-3.5.2.tar.gz
  |-- mysql
      \-- mysql-5.7.11.tar.gz
  \-- numpy
      \-- numpy-1.8.0.tar.gz

It is important to remark that each file could be even a softlink to the existing source files located in a different path.

Finally, Spack has to know about this new mirror, so you need to add it to your current configuration either writing the mirror.yaml file by hand or typing:

spack mirror add name http://myserver.com/packages/newMirror/

Now Spack will look for every package tarfile in this mirror and if not found in the url of the package.

Using LCGPackages mirror

Since we make use of this eos repository [http://lcgpackages.web.cern.ch/lcgpackages/tarFiles/sources] to host the tarfiles of every installed package, we have just created a new folder in the same directory, called spackmirror [LINK], where we follow the directory structure imposed by Spack with softlinks to the original files. This mirror is added to the current configuration with:

spack mirror add name http://lcgpackages.web.cern.ch/lcgpackages/tarFiles/spackmirror/

Adding new repos

By default, Spack provides its own repository with all the officially supported package files. Once cloned the github project, all these recipes can be found in:

./spack/var/spack/repos/builtin/packages/

builtin is how Spack calls the default repository. It's also possible to add different repositories of package files. Similar to mirrors, these repositories have to be organised with a given structure that Spack could understand. The Spack documentation dedicates a section to this matter. Overall, you just need to run:

./spack/bin/spack repo add path/to/your/folder

Hierarchical repos

These new added repositories can be located anywhere in the system. Spack knows where to find them thanks to the configuration file repos.yaml, which by default is in:

./spack/etc/spack/defaults/repos.yaml

Having more than one repository in this file, it is possible to define explicit priorities between them. Spack will look for in first place the package files in the first path, if the repositories doesn't contain that package, Spack will try to find it out in the second path of repos.yaml and so on.

This said, a possible strategy may be to define our own repository before builtin, so that you can overwrite package files for a specific purpose.

repos:
   - $spack/var/spack/repos/testingpackages
   - $spack/var/spack/repos/mypackagefiles
   - $spack/var/spack/repos/builtin

Using CERN repo

As many of the group projects are still being tested to be fully supported by Spack, their package files cannot be uploaded to the official github repository, so we have created our private repository on Gitlab to store them, named cern-spack

This cern-spack contains:

  • Packages that overwrite some of the official github repository ( builtin)
  • Packages developed by the group

This repository should appear in first place in the repos.yaml in use, on top of builtin.

Since HEP has also its own hep-spack, our proposal is to download both repositories and add them to repos.yaml as follows:

# Clone Spack project
git clone https://github.com/LLNL/spack.git

# Enter to the default location of repositories
cd spack/var/spack/repos

# Here we clone hep-spack and cern-spack
git clone https://github.com/HEP-SF/hep-spack.git
git clone https://gitlab.cern.ch/sft/cern-spack.git

# Add first hep-spack as a repository
cd -
./spack/bin/spack repo add spack/var/spack/repos/hep-spack

# Then add cern-spack
./spack/bin/spack repo add spack/var/spack/repos/cern-spack

The configuration file repos.yaml should look like this:

repos:
   - $spack/var/spack/repos/cern-spack
   - $spack/var/spack/repos/hep-spack
   - $spack/var/spack/repos/builtin

Otherwise, it can be also modified by hand to get this specific order.

Using compilers from AFS/CVMFS

Spack look automatically for the installed compilers in the system, but you may need to use compilers installed in a different location. This happens during the most of LCG builds, where compilers are taken from AFS and CVMFS. These compilers can be added to Spack simply running the following command with the path to where the compiler is installed. For example:

spack compiler add /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-centos7/bin

or

spack compiler add /cvmfs/sft.cern.ch/lcg/contrib/gcc/4.9.3/x86_64-centos7/bin

These commands prepare the environment to build Spack packages using any of these compilers, but the command only takes into account the bin folder of the compiler. In case that you need you add the lib or lib64 folder to the LD_LIBRARY_PATH you will need to specify it in the generated compilers.yaml configuration file:

compilers:
- compiler:
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: centos7
    paths:
      cc : /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-cc7/bin/gcc
      cxx : /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-cc7/bin/g++
      f77 : /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-cc7/bin/gfortran
      fc : /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-cc7/bin/gfortran
    environment:
        set:
          LD_LIBRARY_PATH : /afs/cern.ch/sw/lcg/contrib/gcc/4.9.3/x86_64-cc7/lib64
    spec: gcc@4.9.3
    target: x86_64

You can also run spack compiler add or spack compiler find (the first is an alias for the second) with no arguments to force auto-detection. This is useful if you do not know where compilers are installed, but you know that new compilers have been added to your PATH.

Building from a toolchain

By default, Spack can resolve the version to install for each package in different ways:

  • Taking the latest version specified in the package file.

  • Taking the preferred version if defined

    version('3.1.2', '3af29ec06704cbd08d4ba8d69250ae74', preferred=True)
    
  • Installing the version specified by the user in the command line:

    spack install R@3.1.2
    
  • Taking the version specified in the package.yaml configuration file.

This last file can contain some build parameters to apply either to only one package or as a common configuration for every candidate to be installed.

In order to install a complete stack of packages with a specific version (as we do with LCGCMake and toolchain files), we just need to put that information in the packages.yaml using the correct format:

packages:
  all:
    compiler: [gcc, intel, pgi, clang, xl, nag]
    providers:
      blas: [openblas]
      lapack: [openblas]
      mpi: [openmpi, mpich]
      pil: [py-pillow]
  Davix:
    version: [0.6.2]
  R:
    version: [3.2.5]
  ROOT:
    version: [HEAD]
  blas:
    version: [20110419]
  boost:
    version: [1.62.0]
  castor:
    version: [2.1.13-6]
  cmake:
    version: [3.5.2]
  curl:
    version: [7.48.0]
  dcap:
    version: [2.47.7-1]
  ...

Jenkins integration

We currently have two jenkins jobs dedicated to Spack:

None of the two jobs is periodically running. In both cases, we run a version of the jenkins scripts used in LCGCMake, jk-setup.sh and jk-runbuild.sh.

In addition, we are using a provisional python script (runSpack.py) to launch Spack commands and install either a single package or a complete stack.

All together are in charge of the following processes:

  • Prepare the environment depending on the running build node
  • Clone the Spack github repository
  • Clone the additional repos (cern-spack and hep-spack)
  • Set up the configuration files
    • compilers.yaml
    • repos.yaml
    • mirror.yaml
    • packages.yaml
  • Run Spack according to the options of the jenkins job
  • Send the result report to CDash, if specified

CDash monitoring

The preliminary testing guide can be found here

In order to send the results to our CDash server you need to add this option to the install command:

spack install --log-format=cdash <spec>

and then submit the results to CDash:

curl --upload-file <buildxml> <cdash url>/submit.php?project=<projectname>
curl --upload-file <configurexml> <cdash url>/submit.php?project=<projectname>
curl --upload-file <testxml> <cdash url>/submit.php?project=<projectname>

Here is an example of the commands we are using to report some installation results:

spack install --log-format=cdash --log-file ROOT-cdash ROOT
curl --upload-file ROOT-cdash http://cdash.cern.ch/submit.php?project=Spack

List of LCG packages provided by Spack

Using dev3 as a reference, this list shows the packages provided by the group and its state regarding to Spack.


Application Area Projects

Package Spack Support
COOL No
CORAL No
RELAX No
ROOT Yes - builtin & cern-spack
LCGCMT No
HepMC Yes
Geant4 Yes - builtin & cern-spack
DD4hep Yes - cern-spack (SLC6 and CentOS7)

 


External Packages

Package Spack Support
lcgenv No
4suite No
AIDA No
astroid Yes
autoconf Yes
automake Yes
backports Yes
blas Yes
Boost Yes
certifi Yes
cmmnbuild No
configparser Yes
cppgsl No
cycler Yes
decorator Yes
dill No
distribute No
joblib No
jsonc No
jsoncpp Yes
ccache Yes
CLHEP Yes
CouchDB No
cmaketools No
CMake Yes
cmt No
coin3d No
coverage Yes
CppUnit Yes
cx_oracle No
curl Yes
cython Yes
Davix Yes/cern-spack
doxygen Yes
expat Yes
eigen Yes
elasticsearch No
entrypoints Yes
fastjet No
fjcontrib No
fftw Yes
freetype Yes
fplll No
Frontier_Client No
ftjam No
future No
futures Yes
genshi Yes
gmp Yes
gperftools Yes
gtest No
graphviz No
GSL Yes
hadoop Yes
hepdata_converter No
hepdata_validator No
HepPDT Yes
hive No
hspy No
ipython Yes
ipython_genutils Yes
ipykernel Yes
ipywidgets Yes
java No
jemalloc No
Jinja2 Yes
jpype No
jsonschema Yes
jupyter Yes
jupyter_client Yes
jupyter_core Yes
jupyter_console Yes
jupyter_contrib_nbextensions No
jupyter_contrib_core No
jupyter_nbextensions_configurator No
keras No
lapack Yes
lcov No
libaio Yes
libgit2 No
libsvm No
libtool Yes
libxml2 Yes
libxslt Yes
logilabcommon Yes
lxml No
m4 Yes
MarkupSafe Yes
maven Yes
metakernel No
matplotlib Yes
messaging No
mistune Yes
mock Yes
mpfr Yes
mpfi No
mpich2 No
multiprocess No
multiprocessing No
mysql No
mysql_python No
nbconvert Yes
nbformat Yes
networkx Yes
ninja Yes
nose Yes
notebook Yes
numexpr Yes
numpy Yes
omniorb No
openssl Yes
oracle No
pacparser Yes
pandas Yes
pathlib2 Yes
pathos No
pcre Yes
pexpect Yes
pickleshare Yes
pip Yes
pkg_config Yes (pkg-config)
pox No
ppft No
prettytable Yes
processing No
prompt_toolkit No
protobuf Yes
ptyprocess Yes
psutil Yes
py Yes
py2neo Yes
py4j No
pyanalysis No
pydot No
pydot_ng No
pygments Yes
pygraphics No
pygsi No
pylint Yes
pyminuit No
pyparsing Yes
pyqt Yes
pyqt5 No
pytest Yes
pytimber No
pytz Yes
Python Yes
PythonFWK No
python_dateutil Yes
python_gitlab No
pytools No
PyYAML Yes
pyxml No
pyzmq No
QMtest No
Qt Yes
Qt5 No
qtconsole No
qwt No
R Yes
rangev3 No
requests No
rootpy No
root_numpy No
rpy2 No
singledispatch Yes
scikitlearn Yes
scipy Yes
setuptools Yes
simplegeneric Yes
simplejson No
sip Yes
six Yes
sollya No
soqt No
spark Yes
sqlalchemy Yes
sqlite Yes
stomppy No
storm Yes
subprocess32 No
swig Yes
sympy Yes
tbb Yes
tcmalloc No
terminado Yes
theano No
tornado Yes
traitlets Yes
urllib3 No
uuid Yes
vectorclass No
valgrind Yes
VecGeom Yes
vdt No
wcwidth Yes
Vc No
wheel Yes
widgetsnbextension Yes
xapian Yes
XercesC Yes
xqilla No
xrootd Yes
xz Yes
xrootd_python No
yamlcpp Yes
zeromq Yes
zlib Yes

 


GridExternals and binaries

Package Spack Support
libunwind Yes
igprof No
CASTOR Yes/cern-spack
cream No
dcap Yes/cern-spack
dm No
dpm No
epel No
FTS No
FTS3 No
gfal Yes/cern-spack
gfal2 No
gridftp_ifce No
gridsite No
is_ifce No
lb No
lcgdmcommon No
lcginfosites No
lfc No
srm_ifce Yes/cern-spack
voms No
WMS No
neurobayes No
neurobayes_expert No

 


Generators

 

Package Spack Support
lcgenv No
heputils No
mcutils No
syscalc No
madgraph5amc No
lhapdf No
lhapdfsets No
powheg No
feynhiggs No
chaplin No
pythia8 No
sacrifice No
looptools No
vbfnlo No
FORM No
openloops No
njet No
qgraf No
gosam_contrib No
gosam No
thepeg No
herwig No
herwig3 No
hjets No
pythia6 No
agile No
photos No
evtgen No
rivet No
qd No
blackhat No
sherpa No
hepmcanalysis No
mctester No
hijing No
starlight No
herwig No
crmc No
yoda No
hydjet No
tauola No
jimmy No
hydjet No
alpgen No
pyquen No
baurmc No
professor No
jhu No
vincia No
fastnlo_toolkit No

 


Last changes

09-03-2017: Added coin - Only in cern-spack
09-03-2017: Added vectorclass - Only in cern-spack - [PR not planned]
09-03-2017: Added vc - Only in cern-spack - PR pending
09-03-2017: Added vdt - Only in cern-spack - PR pending
08-03-2017: Added nosexcover - Only in cern-spack
03-03-2017: Added fj-contrib - Only in cern-spack
02-03-2017: Added fast-jet - Only in cern-spack
27-02-2017: Added xqilla - Only in cern-spack
27-02-2017: Added yoda - Only in cern-spack
27-02-2017: Added json-c - Only in cern-spack
24-02-2017: Added hive - Only in cern-spack
24-02-2017: Added dm-util - Only in cern-spack
24-02-2017: Added cream - Only in cern-spack
24-02-2017: Added agile - Only in cern-spack
23-02-2017: Added db(BerkeleyDB) - Only in cern-spack
23-02-2017: Added openldap - Only in cern-spack
23-02-2017: Added keras - Only in cern-spack
23-02-2017: Added theano - Only in cern-spack
23-02-2017: Added pydot - Only in cern-spack
23-02-2017: Added python-gitlab - Only in cern-spack
22-02-2017: Added xrootd-python - Only in cern-spack
21-02-2017: Added subprocess32 - Only in cern-spack
20-02-2017: Added py-timber - Only in cern-spack
20-02-2017: Added stomppy - Only in cern-spack
17-02-2017: Added qtconsole - Only in cern-spack
17-02-2017: Added simplejson - Only in cern-spack
17-02-2017: Added requests - Only in cern-spack
16-02-2017: Added pyzmq - Only in cern-spack
16-02-2017: Added pyxml - Only in cern-spack
16-02-2017: Added future - Only in cern-spack
15-02-2017: Added py4j - Only in cern-spack
15-02-2017: Added pathos - Only in cern-spack
15-02-2017: Added pygsi - Only in cern-spack
15-02-2017: Added processing - Only in cern-spack
15-02-2017: Added multiprocess - Only in cern-spack
14-02-2017: Added multiprocessing - Only in cern-spack
14-02-2017: Added dill - Only in cern-spack
14-02-2017: Added ppft - Only in cern-spack
13-02-2017: Added pox - Only in cern-spack
13-02-2017: Added hepdata-converter - Only in cern-spack
13-02-2017: Added hepdata-validator - Only in cern-spack
13-02-2017: Added elasticsearch - Only in cern-spack
13-02-2017: Added urllib3 - Only in cern-spack
13-02-2017: Added distribute - Only in cern-spack
10-02-2017: Added cx_oracle - Only in cern-spack
10-02-2017: Added couchDB - Only in cern-spack
10-02-2017: Added cmmnbuild - Only in cern-spack
10-02-2017: Added lxml - Only in cern-spack
10-02-2017: Added 4Suite - Added to Official Spack Github repo
25-01-2017: Added DD4hep - Only in cern-spack (SLC6 and CentOS7, but not ubuntu yet)
25-01-2017: Added pacparser - Only in cern-spack
24-01-2017: Added Geant4 - Version added to Official Spack Github repo
24-01-2017: Added VecGeom - Version added to Official Spack Github repo
23-01-2017: ROOT can be built in Ubuntu16(native compiler), CentOS7 and SLC6 (gcc49, 62) - Version in cern-spack differs from the one in builtin

You are here