KAIRA data rates

KAIRA is capable of producing different types of data files, depending on the configuration of the software and hardware control parameters. These different data types are suited to different experiments and it is not necessary (or even desirable) to record all of them.

Standard data products

There are standard data products, which are the statistics files generated by the standard LOFAR system. These are typically used for programmes such as riometry, solar observing, scintillation experiments, station calibration, and so on. These standard data products are as follows:

SST -- Subband-Statistics

These record a 1-second average power, for each subband, for each of the 96 receiver units (RCU). This produces 96 files, with N records (where N is the number of seconds), where each record is 512 subbands in size.

BST -- Beamlet Statistics

These are 1-second average power, for each beamlet for each polarisation. There are either 244x16-bit, 488x8-bit or 976x4-bit beamlets. (In reality the 16-bit beamlets only have 12-bits of dynamic range, but are still packed into 16-bit words). This produces two files (X and Y polarisation), with N records (where N is the number of seconds), where each record is B beamlets in size.

ACC -- Array Covariance Cubes

ACC files (also sometimes referred to as Antenna Cross-Correlation) files contain visibility data for all the baseline pairs within the array, averaged to 1 second.

XST -- Cross-correlation Statistics

Similar to ACC files, but these record each second a single subband, rather than stepping through them. Additionally, rather than resetting and starting a new file after 512 seconds, it records continuously. Note that it is not possible to recrod both ACC and XST files at the same time.

Data rates

It is important that observers appreciate how much data KAIRA can record, so that data storage and transport can be appropriately managed. The following table summarises the rates, assuming a standard configuration:

Type	Files per day	Size per file	Daily total	Rate
SST	96	352149504	33.8 GB	391 kB/s
ACC	164	75497472	12.4 GB	143 kB/s
BST	4	167821248	0.7 GB	7.7 kB/s

Non-standard data products

In addition to the standard data products, KAIRA can also record raw voltage level data. These are typically recorded with programmes such as KLP or KUG (but not both!). For these, the data rates can be extremely high and, as such, it does not make sense to indicate a daily total (as that would easily our existing on-site storage).

Type	Files	Individual rate	Total rate
KUG	4	96 MB/s	383 MB/s
KLP	approx. 244/s	90 MB/s	380 MB/s

Please note that any experiment requiring raw voltage-level data will require these non-standard products. This includes Incoherent Scatter Radar (ISR) experiments and Very-Long Baseline Interferometry (VLBI) observations. In all such cases, data storage and handling will need to be coordinated with the KAIRA team. In large data-volume cases, observers will be expected to provide their own storage.

These represent maximum values. Sometimes we decimate the data (pulsar observing) or average it (interplanetary scintillation observations). These data can be taken in addition to standard data products.


References

• "Computing for data acquisition", KAIRA document KAIRA-SGO-PRP-002.
• "LOFAR station data cookbook", ASTRON document LOFAR-ASTRON-MAN-064.