Kaira Local Pipeline

Today we decided to release the code for our local processing pipeline that we have been working on. We didn't have time to invent a fancy name for it, so just decided to call it the KAIRA Local Pipeline (KLP). You can download it from here: http://www.sgo.fi/~j/klp-1.0.tar.gz. The 1.0 release contains three simple processors: file recorder (to store beamlet data on disk), average power recorder (for e.g., interplanetary scintillation measurements), and a null processor that does nothing (for performance testing the udp packet processing).

While there already exists at least one program (PELICAN) with similar capabilities, we wanted a program that is very simple and allows users get going without learning how to use a larger framework. If you need a sophisticated highly configurable object oriented framework for processing LOFAR data, you should probably be considering PELICAN. If you want something simple and ANSI C based, with minimal supporting utilities and configuration files, you might want to check out KLP.

The hardware needed to run KLP is the similar to what one would have for running ARTERMIS (the pulsar processing pipeline developed at Oxford) or any other local processing pipeline on a LOFAR station. At KAIRA we currently only have one Linux PC with a quad gigabit ethernet interface. This quad gigabit ethernet is connected to the four data lanes coming out of the four RSPs that output the beamformed beamlets. For our current purposes, one machine is enough, but in the future, it would also be possible to install KLP on a Linux cluster and use, e.g., MPI for interprocess communication.

Two possible hardware configurations to run the LOFAR local processing. 

As already said, the software architecture is very simple. There is one process for each lane and there are two threads that work in parallel on the incoming data. The first thread reads the udp packets coming in from the LOFAR RSP, parses them and puts them into a double buffer. Another thread waits for the double buffer to flip and once this happens, it then calls the klp_proc() function that processes the contents of the buffer. All of the underlying threading and synchronization is hidden away from the user, who only has to implement klp_proc().

The software architecture detailing all of the inner workings of KLP. The way to implement your own processing is to define klp_proc() and klp_init() and link it with klp_core.c.