I am leaving Google and similar companies cloud services and moving to my own infrastructure for my personal data. This process is going to take a while and I am going to document it here to make it easier for others. However the obvious question is why move from free cloud services which already exist and are easy to use to paying for my own infrastructure and configuring it myself? Well partly I do not want to be the product any more which is being sold, I want to be the customer not merely a user who is being sold to advertisers. Since there is no way to pay Google to stop selling me I have to go elsewhere. I could go to someone like MyKolab which claims to care about privacy and do things properly – and people who cannot roll their own probably should think about it – but I get better guarantees from rolling my own and it should be a good learning experience.

Also Snowden. My aim is to make it such that if anyone (including state actors) want my data, then the easiest way of gaining access to it is to come and ask me nicely, we can discuss it like civilised people over tea and cake and if you make a sensible argument then you can have it. If not come back with a warrant. I am not a criminal or a terrorist and I do not expect to be treated like one with all my communications being intercepted. My data includes other people’s personally identifying information (PII) and so can only be disclosed to people who they would expect it to be given to for the purpose for which it was provided. That does not include GCHQ etc. and so I am not following the spirit of the Data Protection Act (DPA) if I make it possible for other people to obtain it without asking.

Similarly some of my friends work for Christian, environmental, aid or democracy organisations, sometimes in countries where doing so is dangerous. Information which might compromise their security is carefully never committed to computer systems (such operational security has been common in Christian circles for 2000 years) but sometimes people make mistakes, particularly when communicating internally in ‘safe’ countries like the UK. However no countries have clean records on human rights etc. and data collected by the ‘five eyes’ is shared with others (e.g. unfiltered access is given to Israel) and there are countries who are our allies in the ‘war on terror’ but which also persecute (or have elements of their security forces who persecute) minorities or groups within their country. I might in some sense be willing to trust the NSA and GCHQ etc. (because they have no reason to be interested in me) but I cannot because that means trusting 800,000 people in the US alone, some of whom will be working for bad governments.

Similarly while our present government is mostly trying to be good if frequently foolish. It is very easy for that to change. So we need to ensure that the work required to go from where we are to a police state is huge so that we have enough time to realise and do something about it. Presently the distance to cover in terms of infrastructure is far too small, being almost negligible. It is our duty as citizens to grow that gap and to keep it wide.

So I am going to try and find solutions which follow best practises of current computer security, following the principle of least privilege and using compartmentalisation to limit the damage that the compromise of any one component can cause. I am going to document this so that you can point out the holes in it so that we can learn together how to do this properly.

Maybe some of this might even help towards my PhD…

This summer I am working on the Google Summer of Code project “Revive GNU Prolog for Java”. It now has a project page and a Git repository which resulted in a rather entertaining screen shot.

Yesterday I found out that last year someone did a lot of the work I was intending to do this summer and they gave me a svn dump of the changes that they made. I had previously found two other people that had made some changes over the last 10 years while the project was dormant.

So I was faced with the task of taking the SVN dump of the changes made by Michiel Hendriks (elmuerte) and splicing them onto the old CVS history of the code he took the source .zip and then converting it into Git (which is what we are now using as our VCS). I was kind of hoping that with luck the two development histories would then share a common root which could be used to help with merging the two development histories back together that hope was in vain (though I have another idea I might try later).

Anyway this whole splicing thing was non-trivial so I thought I would document how I did it (partly so that I can find the instructions later).

So there were problems with the SVN dumpfile I was given: it couldn’t be applied to a bare repository as all the Node-Paths had an additional extension ‘trese/ample/trunk’ on the beginning. No one fix I tried was to add

Node-path: trese
Node-kind: dir
Node-action: add

Node-path: trese/ample
Node-kind: dir
Node-action: add

Node-path: trese/ample/trunk
Node-kind: dir
Node-action: add

Into the the first commit using vim: now this worked but it meant that I still had the extra ‘trese/ample/trunk/’ which I didn’t want.
So I removed that using vim and “:%s/trese\/ample\/trunk\///g”, unfortunately there were a couple of instances where trese/ample/trunk was refereed to directly when files were being added to svn:ignore unfortunately I didn’t find out how to refer to the top level directory in an svn dump file so I just edited those bits out (there were only 2 commits which were effected). So now I had a working svn dumpfile. To do the splicing I used svndumptool.py to remove the first commit resulting in a dumpfile I called gnuprolog-mod.mod2-86.svn.dump (see below for instructions on how to do this).

I got a copy of the CVS repository, so that the current working directory contained CVSROOT and also a folder called gnuprolog which contained the VCed code.

# This makes a dumpfile 'svndump' of the code in the gnuprolog module I only care about the trunk.
cvs2svn --trunk-only --dumpfile=svndump gnuprolog
# Then we use svndumptool to remove the first commit as cvs2svn adds one to the beginning where it makes various directories.
svndumptool.py split svndump 2 8 cvs-1.svn.dump
# Then we use vim to edit the dump file and do :%s/trunk\///g to strip of the leading 'trunk/' from Node-Paths
vim cvs-1.svn.dump
# Create a SVN repository to import into
svnadmin create gnuprolog-mod.plaster.svn
# Import the CVS history
svnadmin load gnuprolog-mod.plaster.svn < cvs-1.svn.dump
# Import the SVN history
svnadmin load gnuprolog-mod.plaster.svn < gnuprolog-mod.mod2-86.svn.dump
# Make a git repository from the SVN repository
git svn clone file:///home/daniel/dev/gnuprolog/gnuprolog-mod.plaster.svn gnuprolog-mod.plaster.git

Things that I found which are useful

The SVN dump file format
svn-to-git got me the closest to being able to import the svn dumpfile into git.
These instructions on how to fix svn dumpfiles.

Sorry this post is rather ramblely stuck half way between a howto and an anecdote but hopefully someone will find it useful.

I have obtained the source code for JULES which is one of the components that the Met Office uses to forecast the weather. Compiling it was not straightforward and required the modification of several source files.

The modified files and .patch files can be found here.

First I will outline the route I took to the most working binary.

An almost working way

1) in Makefile:

1.1) Setting the location of the netcdf libaries (you will probably need to install these)

JULESDIR=$(PWD)

CDF_LIB_PATH=/usr/lib/

CDF_MOD_PATH=/usr/include/

1.2) and to enable the use of the gfortran compiler

ifeq ($(COMPILER),g95)

include $(JULESDIR)/Makefile.comp.g95

endif

ifeq ($(COMPILER),gfortran)

include $(JULESDIR)/Makefile.comp.gfortran

endif

ifeq ($(COMPILER),nag)

include $(JULESDIR)/Makefile.comp.nag

endif

rather than

ifeq ($(COMPILER),g95)

include $(JULESDIR)/Makefile.comp.g95

endif

ifeq ($(COMPILER),nag)

include $(JULESDIR)/Makefile.comp.nag

endif

2) in Makefile.common.mk:

GRMPATH=$(which grm)

ifneq ($(strip $(GRMPATH)),)
    RM=grm -f
endif

RMPATH=$(which rm)

ifneq ($(strip $(RMPATH)),)
    RM=rm -f
endif

ifndef RM
    $(error No rm equivalent command found)
endif

rather than

RM=grm -f

grm means GNU rm, on a linux system rm is GNU rm, so rm should be used rather than grm the conditionals mean that if the met office run the code on their setup it ‘should work’ and if someone runs it on a normal linux distribution it should still work.

cp Makefile.comp.g95 Makefile.comp.gfortran

3) in Makefile.comp.gfortran:

In order to select the gfortran compiler rather than g95, as gfortran is available for Ubutnu/debian while g95 is not.

FC=gfortran -fno-underscoring

rather than:

FC=g95

and

MOD_PUT=-J

rather than:

MOD_PUT=-fmod=

4) Then to compile

make BUILD=run COMPILER=gfortran CDFDUMMY=true

#BUILD could also be debug or fast.
#CDFDUMMY is needed even though you can install netcdf from the repositories – because if you set it false you get:

/home/daniel/dev/JULES/jules-v2-0/libjules.a(RWERR_MOD.o): In function `__rwerr_mod_MOD_rwerr':
RWERR_MOD.f90:(.text+0xaae): undefined reference to `__netcdf_MOD_nf90_inquire_dimension'
RWERR_MOD.f90:(.text+0xed7): undefined reference to `__netcdf_MOD_nf90_strerror'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_closecdf':
JULES_NETCDF.f90:(.text+0xd): undefined reference to `__netcdf_MOD_nf90_close'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_check_nc_dims':
JULES_NETCDF.f90:(.text+0x2e1): undefined reference to `__netcdf_MOD_nf90_inq_dimid'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_readvar2dreal_ncvector_gswp2':
JULES_NETCDF.f90:(.text+0x566): undefined reference to `__netcdf_MOD_nf90_inq_varid'
JULES_NETCDF.f90:(.text+0x8b1): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0xbb9): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0xeed): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0x1160): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
JULES_NETCDF.f90:(.text+0x1477): undefined reference to `__netcdf_MOD_nf90_get_var_1d_fourbytereal'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o):JULES_NETCDF.f90:(.text+0x178e): more undefined references to `__netcdf_MOD_nf90_get_var_1d_fourbytereal' follow
/home/daniel/dev/JULES/jules-v2-0/libjules.a(JULES_NETCDF.o): In function `__jules_netcdf_MOD_opencdf':
JULES_NETCDF.f90:(.text+0x1c24): undefined reference to `__netcdf_MOD_nf90_open'
collect2: ld returned 1 exit status
make: *** [jules.exe] Error 1

Successful compilation should result in:

gfortran -fno-underscoring -o jules.exe /home/daniel/dev/JULES/jules-v2-0/JULES.o \
	    /home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy/JULES_NETCDF_DUMMY.o \
	    -L/home/daniel/dev/JULES/jules-v2-0  -L/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -J/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -ljules

Alternatively try to use g95 as recommended

1) Install g95

From here: http://www.gfd-dennou.org/library/cc-env/g95/index.htm.en#label-5
g95 is not available in either Ubuntu or Debian repositories – for unknown reasons.

2) follow steps 1.1 and 2 from the instructions for gfortran.

3) try to compile

make BUILD=run COMPILER=g95 CDFDUMMY=true

If you do this there is no possibility of using netcdf prebuilt binaries according to: http://www.unidata.ucar.edu/support/help/MailArchives/netcdf/msg04125.html
This is because they were built for gfortran and so will only work with gfortran.

unfortunately this results in:

g95 -o jules.exe /home/daniel/dev/JULES/jules-v2-0/JULES.o \
	    /home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy/JULES_NETCDF_DUMMY.o \
	    -L/home/daniel/dev/JULES/jules-v2-0  -L/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -fmod=/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/MODS -I/home/daniel/dev/JULES/jules-v2-0/UTILS/netcdf_dummy  \
	    -ljules
/home/daniel/dev/JULES/jules-v2-0/libjules.a(SOILHY7A.o):(.data+0x0): undefined reference to `darcy_'
/home/daniel/dev/JULES/jules-v2-0/libjules.a(SOILHY7A.o):(.data+0x4): undefined reference to `hyd_con__'
make: *** [jules.exe] Error 1

Running Jules

To run jules you will need to

mkdir OUTPUT

after successfully compiling

and then to run a test run using one of the shipped examples:

./jules.exe < point_loobos_example.jin

unfortunately doing this I got:

At line 152 of file SFSNOW7A.f

Fortran runtime error: Array reference out of bounds for array 'snowcanpft', upper bound of dimension 1 exceeded (6 > 5)

Also trying:

./jules.exe < grid_gswp2_example.jin

fails because:

Opening /users/global/rjel/jules/gswp2_files/lsmask_vector.nc

 fileFormat=nc

 ERROR: attempting to use a netCDF procedure, but netCDF library not available.

 This is a 'dummy' procedure.

 To use netCDF, remake model with the 'proper' netCDF library.

as we compiled with CDFDUMMY=true …

Conclusion

It is possible to compile JULES on Ubuntu Linux with a bit of work. However compiling a working version of JULES on Ubuntu Linux may be beyond my current ability (I don’t know any FORTRAN).

Hopefully this will help someone.

Thanks as always go to the denizens of #srcf on irc.srcf.ucam.org

This was harder than expected and I did not find any instructions on the internet – the manual from the CASIO website was useless.

The battery required was a CR 2016 which I got from maplin.

Getting the battery out and putting a new one in was not too hard – taking the back off required a small philips screwdriver and then the central section was levered out and the inner metal cover removed by unclipping it. Putting the new battery in was also similarly easy.

However this did not make the watch work – it was still blank and needed reseting – though the light worked. I could find no way of fixing it and so in the end I held down the light button and poked the copper contacts inside with a pin connecting them to the outer metal case in the hope of flipping some switch somewhere and so making it work – to my surprise this worked.

Note: that there is a little spring inside the case and so be very careful when turning the inside section upside down to keep your finger over the spring to stop it falling out.