Observing tips #5 -- Checking Python versions

There is a lot of software that is used by LOFAR stations which is written in the Python computing language. While it is certainly a well-used and well-established software environment, there are certainly occasions when knowing the exact version of the libraries that you are using is very helpful.

In the case of KAIRA, we also use LOFAR software (and much of our software is conversely applicable to other LOFAR stations). So, there are occasions where we need to check on the version of a given installation.

To do this, we have this small Python programme which we use to check.

#!/usr/bin/python

import sys
import numpy
import scipy
import ephem
import matplotlib

print "Installed versions..."
print "python =",sys.version
print "numpy =",numpy.version.version
print "scipy =",scipy.version.version
print "(py)ephem =",ephem.__version__
print "matplotlib =",matplotlib.__version__

Simply run it, and it will print out the version numbers of the typically used packages: Python itself, NumPy, SciPy, PyEphem and MatPlotLib.

When I ran it just now on one of the KAIRA machines, I read the following:

~> ./swversion.py
Installed versions...
python = 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3]
numpy = 1.6.1
scipy = 0.9.0
(py)ephem = 3.7.5.1
matplotlib = 1.1.1rc
~> 

We hope that helps!

Observing tips #4 — Keeping watch

Often during observing there are tasks that require a certain amount of time to complete. Sometimes these are things that can be simply left to run and at other times you might need to keep an eye on them. Occasionally, there are also occasions when you want to monitor some system parameter on the computer. This can be easy enough to do once, but often it results in typing the command over and over (or using up-arrow / carriage return).

Let's take disk space on the file system, for instance. This is often done with the 'df' command. Now, to be a bit clever, you could put this in a loop. Here is an example (the '%' is just the prompt).

    % while true ; do df ; done

That runs it a bit fast, so we can add some wait time between iterations

    % while true ; do df ; sleep 2 ; done

Better, but still difficult to read. So, we can clear the terminal each time first.

    % while true ; do clear ; df ; sleep 2 ; done

Although the above works, and is effective, there is a simpler way, that exists on most modern Linux systems... watch!

    % watch df

This does exactly what the above command sequence was doing, but it also prints the current time, the update frequency and the command being run. If you want to monitor more complex commands, simply put them in quotes. Such as:

    % watch "ls -l"

To exit from a 'watch', press CTRL+C. There are plenty of options for it too. Just run it with the '--help' flag as:

    %  watch --help

On most versions you can control the update time, highlight differences from one update to the next, control the formatting or even get it to automatically drop out on an error.

All in all, this is a useful utility and it makes keeping tabs on those long data transfers much easier.

Observing tips #3 — Starting/stopping KBT experiments

KBT (KAIRA Background Task) as an observing system for scheduling and running pre-designed scripts. The following is the instruction set for getting KBT observations started.

Starting KBT observations

1.  Be organised and log in to KAIRA01 at least 30 mins in advance, thus checking that you have a password, etc.
2. When you have been allocated the telescope, you will receive an e-mail from someone (usually the previous observer). 
3. When you get the "it's all yours now" e-mail, log in to the LCU.
4. Double check the system is reset and ready to use. To do this, type:
   
   %  swlevel
   
   You should see "Currently set level is 0" and all processes are down. If not, get help.
5. Check the current experiment with kbt.
   
   %  kbt --status
   
   It should be STOPPED. If not, get help.
6. Change the experiment to yours.
   
   % kbt --list
   % kbt --experiment=rio3_v45z
   
7. Start the experiment
   
   % kbt --start
   
   Your experiment is now starting. Note that it takes 2 minutes to fully start. Eventually, you will get your prompt back.
8. Your experiment is now running. You should check that it is recording data files. First, use kbt to find where the data is going:
   
   % kbt --variables
   
   Look for "datapath".
9. Use "ls" to inspect the datapath and check that the data is being recorded.If all is going well, there is no need to give the system further attention until the observations are over.

Stopping KBT observations

1. Stop your observations with KBT.
   
   %  kbt --stop
   
   Note that this might take a little while.
2. Double check that the system is reset and ready to use for the next observer. To do this, type:
   
   %  swlevel
   
   You should see "Currently set level is 0" and all processes are down. If not, get help.
3. Then, and this is VERY IMPORTANT, send an e-mail to the next observer with a CC to lofar-obs to inform the observing team that you are finished.

That's it! You should now consider exporting your data.

Observing tips #2 — Keeping an observing log

In the old days, we all used to keep observing logs. It would have the date and time of all the things that we did, what we were looking at and any notes that we thought were important along the way. It was in a log book; yes, a physical book and the notes would all be hand-written.

Things have moved on, of course, and these days the telescope control, the data acquisition and the data processing are all done on computers. While some of us still keep log books, it is all too easy to forget to write down a command or two... usually the critical one that you need to remember later (because it had lots of arguments!).

As most of our systems use Linux (or some other UNIX-like environment), there is the "history" command, which can be used to see all the commands that have been put into the system. This is very useful. For example:

   % history

    1  ls

    2  ls - ltr
    3  cat obs_notes.txt
    4  history

But by default it does not include the time when the command was issued. However, there is a way around this. In bash (a common shell used on many Linux systems), the history command will check for the history format shell variable. So, you can set this before running your history command (in fact, it might be useful to put it in your startup script, such as .bashrc).

   % export HISTTIMEFORMAT="%Y%m%d-%H%M%S %% "

Then, when you run your history command, you get this:

   % history

    1014  20120827-114212 % swlevel 3
    1015  20120827-114218 % rspctl --regstat
    1016  20120827-114347 % poweruphba.sh 5
    1017  20120827-114410 % rspctl --rcu
    1018  20120827-114422 % cd

I prefer to use the compact format "YYYYMMDD-HHMMSS", because that way you can sort on this field. While this may seem superfluous for a simple example like this, what may happen is that there are commands coming in for all sorts of different terminals. So, what you can then do is to save them all, and then sort by time.

For example, in the first terminal:

   term1% history >> obs_notes.txt

And then, in the second terminal:

   term2% history >> obs_notes.txt

   term2% sort -k 2 obs_notes.txt

The sort command will then put them all into order for you. Note that you need to use ">>", not ">" in order to append to the obs_notes.txt file. Otherwise you simply overwrite the previous version.

Of course, you can play around with awk and grep and sed to get other fancy formatting, but you get the idea.

Hope it helps!

Observing tips #1 — Introduction

There are three main observing modes for LOFAR stations:

• Network mode — where the station is controlled from Dwingeloo as part of the overall LOFAR network.
• Single-station mode —  when the station is controlled from Dwingeloo, but not as part of the network.
• Stand-alone mode — when the station is controlled completely locally.

For KAIRA, this will mostly be in stand-alone mode. As a result we will be doing a lot of observing and will gain a lot of experience with using the LOFAR systems directly.

We've noticed that as we've been doing our work we've picked up some useful tips that would apply to any of  the other LOFAR international stations. These are mostly command-line tricks that can be used on the local control unit, but also for some of the other data processing aspects of the system. Hopefully these will be of use to other LOFAR users and even other scientists running their own experiments, even on other equipment. And, for our general readers, it will give a small glimpse of some of the strange commands that we use to interact with our experiments.

We shall be occasionally interspersing these amongst the photographs and results that we post here. Watch out for them, or search for them all with the observing tips label.