Posts Tagged ‘linux’

Raspberry Pi Entropy server

Thursday, August 23rd, 2012

The Raspberry Pi project is one of the more popular projects the Computer Lab is involved with at the moment and all the incoming freshers are getting one.

One of the things I have been working on as a Research Assistant in the Digital Technology Group is on improving the infrastructure we use for research and my current efforts include using puppet to automate the configuration of our servers.

We have a number of servers which are VMs and hence can be a little short of entropy. One solution to having a shortage of entropy is an ‘entropy key‘ which is a little USB device which uses reverse biased diodes to generate randomness and has a little ARM chip (ARM is something the CL is rather proud of) which does a pile of crypto and analysis to ensure that it is good randomness. As has been done before (with pretty graphs) this can then be fed to VMs providing them with the randomness they want.

My solution to the need for some physical hardware to host the entropy key was a Raspberry Pi because I don’t need very much compute power and dedicated hardware means that it is less likely to get randomly reinstalled. A rPi can be thought of as the hardware equivalent of a small VM.

Unboxed Raspberry Pi with entropy key

I got the rPi from Rob Mullins by taking a short walk down the corridor on the condition that there be photos. One of the interesting things about using rPis for servers is that the cost of the hardware is negligible in comparison with the cost of connecting that hardware to the network and configuring it.

The Raspberry Pi with entropy key temporarily installed in a wiring closet

The rPi is now happily serving entropy to various VMs from the back of a shelf in one of the racks in a server room (not the one shown, we had to move it elsewhere).

Initially it was serving entropy in the clear via the EGD protocol over TCP. Clearly this is rather bad as observable entropy doesn’t really gain you anything (and might lose you everything). Hence it was necessary to use crypto to protect the transport from the rPi to the VMs.
This is managed by the dtg::entropy, dtg::entropy::host and dtg::entropy::client classes which generate the relevant config for egd-linux and stunnel.

This generates an egd-client.conf which looks like this:

; This stunnel config is managed by Puppet.

sslVersion = TLSv1
client = yes

setuid = egd-client
setgid = egd-client
pid = /egd-client.pid
chroot = /var/lib/stunnel4/egd-client

socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
TIMEOUTclose = 0

debug = 0
output = /egd-client.log

verify = 3

CAfile = /usr/local/share/ssl/cafile

[egd-client]
accept = 7777
connect = entropy.dtg.cl.cam.ac.uk:7776

And a host config like:

; This stunnel config is managed by Puppet.

sslVersion = TLSv1

setuid = egd-host
setgid = egd-host
pid = /egd-host.pid
chroot = /var/lib/stunnel4/egd-host

socket = l:TCP_NODELAY=1
socket = r:TCP_NODELAY=1
TIMEOUTclose = 0

debug = 0
output = /egd-host.log

cert = /root/puppet/ssl/stunnel.pem
key = /root/puppet/ssl/stunnel.pem
CAfile = /usr/local/share/ssl/cafile

[egd-host]
accept = 7776
connect = 777
cert = /root/puppet/ssl/stunnel.pem
key = /root/puppet/ssl/stunnel.pem
CAfile = /usr/local/share/ssl/cafile

Getting that right was somewhat tedious due to defaults not working well together.
openssl s_client -connect entropy.dtg.cl.cam.ac.uk:7776
and a python egd client were useful for debugging. In the version of debian in rasperian the stunnel binary points to an stunnel3 compatibility script around the actual stunnel4 binary which resulted in much confusion when trying to run stunnel manually.

LaTeX user search path

Monday, November 1st, 2010

Because this took far too long to find out:
If you have .sty files you have written want to be able to easily reuse in multiple documents then put them in ~/texmf/tex
Then you should be able to use them with \usepackage{foo} as normal.

Compiling JULES on Ubuntu Intrepid Linux

Thursday, February 5th, 2009

I have obtained the source code for JULES which is one of the components that the Met Office uses to forecast the weather. Compiling it was not straightforward and required the modification of several source files.

The modified files and .patch files can be found here.

First I will outline the route I took to the most working binary.

An almost working way

1) in Makefile:

1.1) Setting the location of the netcdf libaries (you will probably need to install these)

JULESDIR=$(PWD)
CDF_LIB_PATH=/usr/lib/
CDF_MOD_PATH=/usr/include/

1.2) and to enable the use of the gfortran compiler

ifeq ($(COMPILER),g95)
include $(JULESDIR)/Makefile.comp.g95
endif
ifeq ($(COMPILER),gfortran)
include $(JULESDIR)/Makefile.comp.gfortran
endif
ifeq ($(COMPILER),nag)
include $(JULESDIR)/Makefile.comp.nag
endif

rather than

ifeq ($(COMPILER),g95)
include $(JULESDIR)/Makefile.comp.g95
endif
ifeq ($(COMPILER),nag)
include $(JULESDIR)/Makefile.comp.nag
endif

2) in Makefile.common.mk:

GRMPATH=$(which grm)

ifneq ($(strip $(GRMPATH)),)
    RM=grm -f
endif

RMPATH=$(which rm)

ifneq ($(strip $(RMPATH)),)
    RM=rm -f
endif

ifndef RM
    $(error No rm equivalent command found)
endif

rather than

RM=grm -f

grm means GNU rm, on a linux system rm is GNU rm, so rm should be used rather than grm the conditionals mean that if the met office run the code on their setup it ‘should work’ and if someone runs it on a normal linux distribution it should still work.

cp Makefile.comp.g95 Makefile.comp.gfortran

3) in Makefile.comp.gfortran:

In order to select the gfortran compiler rather than g95, as gfortran is available for Ubutnu/debian while g95 is not.

FC=gfortran -fno-underscoring

rather than:

FC=g95

and

MOD_PUT=-J

rather than:

MOD_PUT=-fmod=

4) Then to compile

make BUILD=run COMPILER=gfortran CDFDUMMY=true

#BUILD could also be debug or fast.
#CDFDUMMY is needed even though you can install netcdf from the repositories – because if you set it false you get:

/home/daniel/dev/JULES/jules-v2-0/libjules.a(RWERR_MOD.o): In function `__rwerr_mod_MOD_rwerr':
RWERR_MOD.f90:(.text+0xaae): undefined reference to `__netcdf_MOD_nf90_inquire_dimension'
RWERR_MOD.f90:(.text+0xed7): undefined reference to `__netcdf_MOD_nf90_strerror'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_closecdf':
JULES_NETCDF.f90:(.text+0xd): undefined reference to `__netcdf_MOD_nf90_close'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_check_nc_dims':
JULES_NETCDF.f90:(.text+0x2e1): undefined reference to `__netcdf_MOD_nf90_inq_dimid'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_readvar2dreal_ncvector_gswp2':
JULES_NETCDF.f90:(.text+0x566): undefined reference to `__netcdf_MOD_nf90_inq_varid'
JULES_NETCDF.f90:(.text+0x8b1): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0xbb9): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0xeed): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0x1160): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0x1477): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o):JULES_NETCDF.f90:(.text+0x178e): more undefined references to `__netcdf_MOD_nf90_get_var_1d_fourbytereal' follow
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_opencdf':
JULES_NETCDF.f90:(.text+0x1c24): undefined reference to `__netcdf_MOD_nf90_open'
collect2: ld returned 1 exit status
make: *** [jules.exe] Error 1

Successful compilation should result in:

gfortran -fno-underscoring -o jules.exe /home/daniel/dev/JULES/jules-v2-0/JULES.o \
	    /home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy/JULES_NETCDF_DUMMY.o \
	    -L/home/daniel/dev/JULES/jules-v2-0  -L/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -J/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -ljules

Alternatively try to use g95 as recommended

1) Install g95

From here: http://www.gfd-dennou.org/library/cc-env/g95/index.htm.en#label-5
g95 is not available in either Ubuntu or Debian repositories – for unknown reasons.

2) follow steps 1.1 and 2 from the instructions for gfortran.

3) try to compile

make BUILD=run COMPILER=g95 CDFDUMMY=true

If you do this there is no possibility of using netcdf prebuilt binaries according to: http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg04125.html
This is because they were built for gfortran and so will only work with gfortran.

unfortunately this results in:

g95 -o jules.exe /home/daniel/dev/JULES/jules-v2-0/JULES.o \
	    /home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy/JULES_NETCDF_DUMMY.o \
	    -L/home/daniel/dev/JULES/jules-v2-0  -L/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -fmod=/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -ljules
/home/daniel/dev/JULES/jules-v2-0/libjules.a(SOILHY7A.o):(.data+0x0): undefined reference to `darcy_'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(SOILHY7A.o):(.data+0x4): undefined reference to `hyd_con__'
make: *** [jules.exe] Error 1

Running Jules

To run jules you will need to

mkdir OUTPUT

after successfully compiling

and then to run a test run using one of the shipped examples:

./jules.exe < point_loobos_example.jin

unfortunately doing this I got:

At line 152 of file SFSNOW7A.f
Fortran runtime error: Array reference out of bounds for array 'snowcanpft', upper bound of dimension 1 exceeded (6 > 5)

Also trying:

./jules.exe < grid_gswp2_example.jin

fails because:

Opening /users/global/rjel/jules/gswp2_files/lsmask_vector.nc
 fileFormat=nc
 ERROR: attempting to use a netCDF procedure, but netCDF library not available.
 This is a 'dummy' procedure.
 To use netCDF, remake model with the 'proper' netCDF library.

as we compiled with CDFDUMMY=true …

Conclusion

It is possible to compile JULES on Ubuntu Linux with a bit of work. However compiling a working version of JULES on Ubuntu Linux may be beyond my current ability (I don’t know any FORTRAN).

Hopefully this will help someone.

Thanks as always go to the denizens of #srcf on irc.srcf.ucam.org