REDUCEIR - CLASSIC/CLIMB data reduction software - HOWTO -------------------------------------------------------- Theo ten Brummelaar - 2012-01-06 This is a brief outline on how to use the data reduction package known as reduceir to analyze data taken with CHARA CLASSIC. More detailed documentation will be forth coming.... eventually. The technique for obtaining visibilities from CLASSIC data is described in the first two CHARA publications. 1. Installing the software. Binaries are distributed for 32 and 64 bit Linux and for Mac OS X 10.5.6. Bundled tar files are diustributed using my Dropbox account. Email me for details. The software should compile on any Unix machine, and requires a working copy of GTK and Gnuplot. There is a README file in the distribution that describes how to install the software. Note that the Mac version uses the jhbuild environment. See http://www.gtk.org/download/macos.php 2. What is contained in the distribution The follow files are in the software package: ampvir - Visibility estimator based on fringe amplitude and fringe fitting. manbinir - Manual fitting of separated fringe packets. breakir - Break a file up separating shutter sequences and data calcuvb - Calculate the UV point and baseline of a data file. calcvir - Calculate Visibility based on fringe power spectra. editir - Edit a data file. extractir - Write a user selected text summary of a set of output files. calibir - Calibrate visibilities and write an OIFITS file. findfrgir - Find fringes in a data file. findmjd - Write a list of names and MJDs for a set of observations. psir - Calculate the power spectra of a data file. summaryir - Write a predefined text summary of a set of output files. truncateir - Truncate saturated fringe scans. reduceir - GTK based GUI wrapper for reduction software. chara_plan - Plan an observation. astromod - Run the astrometric model. This program is now mostly obsolete. chara_plan2 - New version of planning tool for more than two telescopes. fakeir - Create a fake data file in the old format. fakecc - Create a fake data file in the new FITS format. fakebin - Create a fake separated fringe packet (SFP) data file. delpos - Calculate the delay line positions for a star. ccbrief - Read an CLASSIC FITS file and print out the data as text. irbrief - Read a older CLASSIC data file and print out the data as text. fakeud - Create fake observations simulating a Uniform Disk observation. oifud - Fit a Uniform Disk, or Limb Darkened disk to an OIFITS file. vis2.fit - Example OIFITS file. starlist - List of CHARA targets. telescopes.chara - Data on CHARA telescopes. system.chara - Data on CHARA system. environ - Data on CHARA environment. redclimb - CLIMB data reduction pipeline. Most of these programs are command line run and have switches of the normal Unix type. For "normal" CLASSIC data reduction there is a GTK based wrapper you can use so you don't need to run the commands manually. If you wish to process a large amount of data the command line routines can be used in scripts. 3. Getting the data. First, please don't puddle in the data archives, copy your data to a local directory of your own, or set the output directory in the reduceir program to a local directory of your own. The data are kept in our archive system on both michelson.chara-array.org and pease.chara-array.org in the directories /archive/top_bay/CHARA_ARRAY_DATA_ARCHIVE/on_sky /archive/bot_bay/CHARA_ARRAY_DATA_ARCHIVE/on_sky while data that has not yet been archived is contained in /ctrscrut/archive/on_sky on the michelson machine. Please make a compressed archive of your data before downloading it, and please don't download large amounts of data while people are trying to observe. 4. Running reduceir Once you have your data in a local directory, move into that directory and run the GTK wrapper program reduceir. This program has the following flags: usage: reduceir [-flags] Flags: -D[Dir] Directory for results (Basename) -e[scroll_length] Size of scroll window (200) -f Toggle FITs file format (OFF) -R Toggle remote mode (OFF) -h Print this message The scroll length can be changed to a smaller size if the program doesn't fit on your screen. So for example, if you wish the output files to go to the directory /foo/bar and you want a smaller scroll length type: reduceir -D/foo/bar -e100 & The reduceir program is set out as a series of steps you need to work through in order to reduce fringe data arranged from top to bottom. There are many options for each step, but in most cases you can use the defaults. Each program works by adding things to a text file called an "info" file, in which it records all parameters used. This way, when you run reduceir multiple times it will remember where you were up to and which parameters you have used. These info files will also contain the final visibility estimators and are read by the calibration routines and other summary programs like summaryir. Reduceir will warning you if you are repeating a reduction stage, and ask if you wish to continue. You can use the YES and NO buttons at the bottom to reply. The VERBOSE mode is mostly for debugging and the REMOTE mode simply makes all plots smaller. The LOCK SETTINGS button will force all settings to stay the same for the reduction of a sequence of data. You don't need to use reduceir, you can run each step from a command line or even from a script file. 5. Data reduction steps. FILE/DIRECTORY SELECTION Here you select a file to reduce, select an output directory (the default being the current directory) and toggle between using the FITS format or the older data format. BREAKIR This breaks the file up into separate files for the shutter sequences and fringe scans. It will create a directory with the same name as the data file and all results, including the info file, are placed in this directory. The output files are by default in a binary format, but you can choose a text format. Normally the binary format is fine. In most cases you need only click on the BREAKIR button. Note that the terms "BIN" and "SWAP" are dated. This is obsolete so use "BIN". In some cases you may have to force the system to think of the data as Object or Calibrator. You can also manually override the wavelength, CHARA # and other things if you really need to, but this is very rare. TRUNCATE When data is collected in non-destructive or "SYNC" mode it can sometimes saturate the camera near the end of the scans. This routine will plot the mean, and minimum of all scans. You can then click in the window to select the non-saturated part of the fringe scan. The rest of the data will be removed. Having saturation in the data causes a sharp fall off of counts to zero, and this is dangerous when calculating the power spectra as it will add a great deal of noise. FINDFRG Find the fringe positions in the data. Just click on this button. EDITIR Edit the fringe scans. This will allow you to remove scans that do not contain fringes. This is the trickiest part of data reduction. Be careful not to remove too many scans. For example, sometimes seeing makes the fringes disappear, and these scans should be left in place or you will mess up the statistics of the final data product. This normally looks like fringes slowly disappearing and coming back. Places where fringes suddenly disappear for many scans normally mean that the delay is wrong and can be removed. The options are: REDRAW - redraw the current data set SET CURSORS - Place two cursors within the data set. You will need to click twice in the fringe window and green lines should appear. Cursors are always forced to be at fringe boundaries. MAX CURSORS - Set the cursors to the full extent of the data. MAX LEFT - Put the left cursor at the beginning of the data set. MAX RIGHT - Put the right cursor at the end of the data set. ZOOM IN - Zoom in on the data at the current cursor positions. If no cursors are set it will zoom in by 50%. OUT - Zoom out by 50%. << - Move the current zoom window towards the beginning of the file. >> - Move the current zoom window towards the end of the file. SHOW ALL - Zoom out so you can see the entire data set. SHOW SCAN ENDS - Put blue lines at fringe scan boundaries. This is very useful when you have zoomed in and need to know where fringes are within each scan. DEL FRNG - Delete a single scan by clicking somewhere within the scan. DEL SECT - Delete all scans between the current cursors. SAVE FILE - Save the file. QUIT NO SAVE - Quit without saving the data. SAVE & QUIT - Quit but save the data first. Note that at this time there is no undo function. If you screw up, quit without saving and run the editor again. I find a useful technique is to zoom in so that you see about a dozen scans, turn on the SHOW SCAN ENDS function and use the << and >> buttons to move through the data set. PSIR Calculate the power spectra of noise and fringes. You can safely ignore the options and just click on the button. When done the routine will plot the fringe power spectra with the noise removed. This is the best diagnostic of the data quality. The power spectra should be close to zero at the edges and the peak power of the three plots (one for each detector output and one for the difference signal) should line up. Poor power spectra are an indication of poor noise subtraction or poor alignment of the beam combiner. You can play with the plotting parameters and look at just the noise, or a single output, adjust the plot range and so on. AMPVIR Calculate the fringe amplitude using various methods like fringe fitting, envelope calculation, maximum amplitude and so on. Most of the options here are for separated fringe packet analysis which I will not cover here. CALCVIR Calculate the visibility from the power spectra. The most important thing here is to ensure the integration range is set to cover the full width of the fringe power. The first time this is run it will try and automatically select the integration range. It is best, however, to manually select a range by inspecting the power spectra itself. DO OBJECT Lock all settings and run the full data reduction pipeline on all files of the same object. DO DIRECTORY Lock all settings and run the full data reduction pipeline on all files in the directory. Note that it is extremely important to choose an integration range suitable for all files. Since seeing can change it's a good idea to manually reduce data, at least to the power spectrum stage, near the beginning, the middle and the end of a data set. Get a feel for how the fringe power changes width as the seeing changes. The final integration range should include all calibrators and objects of a particular data run. 6. Using the reduced data. Once you have reduced the data you will find a series of new directories, one for each observation, each containing an info file. There are several programs that allow you to inspect these results. The most useful of these is summaryir, which will spit out a summary of all data files sorted by modified Julian date. It outputs various visibility estimates, which should all basically agree. If they don't there is a problem. The program extractir let's you choose which parameters you wish to see. The final step of data reduction is calibration, a function performed by the program calibir, that you must run from the command line. This program has the following flags: usage: calibir [-flags] {OBJ CAL1,CAL2,CAL3...} Flags: -b[beta] Set intensity ratio (Cal/Obj) (1.0) -B[Vis Type] Set Vis estimator for Object and Calibrator (V_LOGNORM) -c Use CHARA number for identifier (OFF) -C[Cal Vis Type] Set Vis estimator for Calibrator (V_LOGNORM) -d Use change in calibrator for error (OFF) -f[oif] Set OIFITS filename (From object) -F Toggle saving OIFITS file (OFF) -h Print this message -H Use HD number for identifier (OFF) -i Use ID/Name for identifier (OFF) -I[12|23|31] Select CLIMB baseline (None) -n Use standard error of mean for error (OFF) -O[Obj Vis Type] Set Vis estimator for Object (V_LOGNORM) -r Print raw data (ON) -s[diam1,diam2,...] Size of calibrators in mas (0.0) -S Self Calibrate mode (OFF) -v Verbose mode (OFF) Visibility estimators available are: V_CMB V_FIT V_ENV V_MEAN_ENV_PEAK V_MEAN_ENV_FIT BINARY_V_A BINARY_V_B BINARY_ENV_V_A BINARY_ENV_V_B V2_SCANS V_SCANS V_NORM V_LOGNORM The first step is normally to run it to display the data: calibir This prints out a summary of the data set sorted by MJD. If it rejects a data file for some reason it will tell you why with an error message. You can run a calibration by choosing a calibrator and an object, for example calibir OBJ CAL1 will set the observations labeled as CAL1 as calibrator and those labeled as OBJ as object. You can have as many calibrators as you like, for example calibir OBJ CAL1 CAL2 The final output is an OIFITS file which is only saved if you set the -F flag. You can also select the calibrators and object by HD number calibir -H 193321 193322 of identifier (-i) or chara number (-c). There are many estimators you can select, the default being LOGNORM. You should use the same estimator for both the calibrator and the object. The estimators available are: V_CMB - Peak of fringe of all scans. V_FIT - Mean of amplitude of fringe function fitted to all scans. V_ENV - Mean of peaks of envelopes. V_MEAN_ENV_PEAK - Peak of mean envelope. V_MEAN_ENV_FIT - Fit of sinc function to mean envelope. BINARY_V_A - Amplitude of fringe fit to first star in SFP data BINARY_V_B - Amplitude of fringe fit to second star in SFP data BINARY_ENV_V_A - Peak of envelope of first star in SFP data BINARY_ENV_V_B - Peak of envelope of second star in SFP data V2_SCANS - V^2 estimate, similar to most other interferometry data. V_SCANS - Square root of the above, not normally a good estimator. V_NORM - Visibility amplitude assuming Gaussian statistics. V_LOGNORM - Visibility amplitude assuming log/normal statistics. Normally one would use V2_SCANS, V_NORM or V_LOGNORM. The first of these will produce an OIFITS file containing a "VIS2" table, and all others an OIFITS file containing a "VIS" table with fringe phase set to zero. This routine works by search forward and backwards in time for each object and finding the nearest calibrator on each side. When you use the -r option it will print out the time difference between the two calibrators in minutes. Not that at present this routines insists on having a calibrator before and after the object in every case. Note that it is very important to give a good estimate of the calibrator diameter. If none is given it will assume the calibrator is not resolved. It's worth fiddling with the dirrerent error estimate methods. I find using the change in calibrator usually gives the most realistic results. When you have something you like use the -F flag to save an oifits file of the result. 7. Calculating a diameter At the present there is only a simple disk fitting routine for an OIFITS file called "oifud". This will plot the data, allow you select which data to include in the fit, and select various models for the object. These models are: Uniform Disk - Simple Uniform Disk. Gaussian - Simple Gaussian. LD Michelson/Pease - Limb Darkened disk using the model of Michelson and Pease (ApJ Vol 53 page 256 1921). This requires a single parameter 'p' representing the power law. Intensity = (D^2 - r^2)^p where D is the diameter and r the position. LD Linear - Linear Limb Darkening, single parameter p. Intensity = 1.0 - p * (1 - sqrt(1 - r^2/D^2)) LD Power Law - Power Law Limb Darkened, See AA Vol 327 page 199 1997 Intensity = sqrt(1.0 - r^2/D^2)^p LD Hanbury Brown - Limb Darkening defined by HB: MNRAS Vol 167 page 121 1974 Intensity = 1.0 - p * (1 - cos(r/D)) 8 - Reduction of CLIMB data The CLIMB data reduction package is called redclimb, and has the following syntax and flags: usage: redclimb [-flags] ir_datafile Flags: -a Toggle apodize for FFT (OFF) -A Use only shutter sequence A (OFF) -B Use only shutter sequence B (OFF) -b Toggle redo background and beams (OFF) -c Toggle pixels must agree (ON) -C Toggle confirm (ON) -d[0,1,2] Set display level(1) -D[Dir] Directory for results (Basename) -e Toggle edit scans (ON) -f[width_frac] Envelope width fraction (0.35) -g[0-1] Fraction of Gaussian to include in CP (0.1) -F Toggle filtering of signal (ON) -h Print this message -i Toggle manual integration range (OFF) -I[12-12,23-23,31,31] Set integration range -m Toggle save means as text (ON) -M Toggle manual data selection in scan (OFF) -n Toggle use noise instead of signal (OFF) -Nndata Force data segment size (AUTOMATIC) -o Toggle skip overlap test (OFF) -p Toggle plot closure phases (ON) -P[smooth_noise_size] Change PS +-smooth size (4) -R Toggle remote mode (OFF) -r Toggle save raw data as text (OFF) -s[n] Scans to skip after shutter change (0) -S[smooth_size] Change +-smooth size (1) -t[start,stop] Truncate scans (OFF) -u Toggle use dither freqs for fringes (OFF) -U[freq] Set DC suppression frequency (10.0 Hz) -v Toggle verbose mode (OFF) -w[weight] Set minimum fringe weight (1.0) -W[width] Set PS peak width for fit (10.0 Hz) -z[pixmult] Set pixelk multiplier (2) This is not a simple program and I have yet to fully convince myself of what the correct defaults should be. A simplified version of what the software does is: - Calculate the background from the shutter sequence and subtract it from the data. - Calculate the amount of light from each beam from the shutter sequence. - From this calculate the various visibility transfer functions - Calculate the mean power spectra for each beam as well as the fringe scans. - From this calculate the predicted noise power spectrum and subtract it from the fringe power spectrum. - From the fringe power spectra work out which frequencies contain fringe information. Note that the default is to use information from the FITs file header for this, but it can be calculated from the data also. - Allow the user to edit each baseline fringe signals. - Use the same integration method as CLASSIC does to estimate visibilities for each baseline. - Calculate the closure phase signal. More on this below. Like the CLASSIC code, redclimb creates a new directory to contain all it's output for each data file and you will find an info file there, as well as the several other output files of interest. The fringe amplitudes are calculated in the same way as calcvir above and the results are placed in the info file. Each one will have the same token name as for CLASSIC except it will also contain a _12, _23, or _31 suffix. The calibir program can be used on these info files by using the -I12, -I23, or -I31 switches. Each will produce an OIFITs file which can then be put together using the oifits-merge program. The software calculates the closure phase in the following way. The location of the fringes for each baseline are calculated. If there are fringes on baselines 23 and 31, the fringes for 12 are assumed to be there and in between the other two baselines. The overlap is then worked out, and if they do not overlap by the selected envelope fraction the scan is rejected. Note that so far I have found that this means most of the scans are rejected, and I now believe that the default 200 scans is not enough, what we probably need is at least 100 scans with good overlap. I will probably change the data collection code to reflect this and also add an on-line closure phase estimate. The area of overlap is then broken up into slices of data, each containing the number of samples over which the lowest frequency fringe has one complete cycle. This is normally 3 times the selected "samples per fringe", whose default is now 5 meaning this is most often 15 samples. This can be longer than the atmospheric time constant so I now recommend using 4 or even 3 samples per fringe for CLIMB. Each of these segments is Fourier transformed and the phase of the 1,2 and 3 cycles per segment are used to calculate a triple product. These are averaged through the data file for the two pixels that see all three baselines, yielding a final closure phase weighted by fringe amplitude. There are way too many factors of -1 and PI in this calculation, but I am now fairly sure I got this right. Pixel 3 often has more noise than Pixel 2 and I have yet to work out why this is. You will find text files containing the triple produces and closure phase information in the data directory. A final warning - the code knows about which telescope triangle is represented in the data, so the sign of the closure phase signal is forced to be +ve clockwise from above. I'm not sure myself if +ve is clockwise or anticlockwise around the triangle as view from above. If some can tell me about this I will change the code to do the right things with closure phase sign. A brief explanation of the flags follows: -a Toggle apodize for FFT (OFF) Add apodization to the FFT routine. This doesn't seem to help much. -A Use only shutter sequence A (OFF) If you missed the second shutter sequence or it is contaminated for some reason. -B Use only shutter sequence B (OFF) Ditto for first shutter sequence. -b Toggle redo background and beams (OFF) If you reprocess a file, as much as possible is pulled out of the info file. This flags forces it to recalculate things. -c Toggle pixels must agree (ON) Will insist both pixels agree on fringe locations within envelope fraction -C Toggle confirm (ON) Will ask you to confirm things by default. -d0,1,2 Toggle display (1) For d=0 you will see no plots, except the final closure phase plot. For d=1 you will see the following plots: - Photometry of each pixel. This is good for checking if the shutter events all work. - The mean noise subtracted power spectra for each pixel. - The pixel 2&3 power spectra with a Gaussian fitted to each fringe signal. These are used to determine the fringe frequency and integration widths. - Waterfall plots for each baseline. You can edit these with the keys: m - Move on to next baseline c - Clear all edits e - Edit r - Redraw You won't see this if you have the -e flag on. - Power spectra for each fringe signal including photometric corrections. Integration ranges are also shown. If you have selected -i you can change the integration range. - Plot of closure phases. For d=2 you will also see the following plots: - Mean power spectra of shutter events and data. - Mean fringe signal with no noise subtraction. - Power spectra of each scan and a line showing where the code thinks the fringe center is. There will be many of these. - Each fringe signal. - If a closure phase calculation is done, the raw or filtered combined fringes, the triple amplitude and the closure phases across the scan. There will be many of these. -D[Dir] Directory for results (Basename) Force a location for the output. -e Toggle edit scans (ON) Turns off the editing. Editing is done for each baseline by showing a waterfall plot on which you can remove segments by clicking in appropriate places. -f[width_fraction] Envelope width fraction (0.35) Fraction of envelope width allowed to still consider fringe envelopes overlap. I'm not sure the default is right, so feel free to experiment. -F Toggle bandpass filtering off. This only affects the displays -g[0-1] Set fraction of closure signal to include. The code fits a Gaussian to the triple product amplitude and only the part of that Gaussian larger than this fraction is included in the averaging. -h Print a list of these flags. -i Turn on manual editing of the integration ranges. -I Allows you to set the integration ranges. This is a good way to ensure that calibrator and object both use the same range. -m Toggle save means as text (ON) Will save the power spectra in text files for your viewing pleasure. -M Toggle manual fringe editing (OFF) This is useful if there clouds get in the way for a while. It also allows you to choose if the A and/or B shutter events are used. -n Toggle use noise instead of signal (OFF) Will use parts of the scans away from the fringes. A good test as the closure signal should be all over the place compared to the fringes. -Nndata Force segment size (automatic). Segment size should be a multiple of 3. -o Toggle skip fringe overlap test (OFF) -p Turn off final plot of closure phases. -Psize Set size of smoothing for noise power spectra (4) -R Toggle remote mode (OFF) Will make the display smaller for remote operation. -r Toggle save raw data as text (OFF) Does what it says, if you wish to have the raw data as a text file. -s[skip] Scans to skip after shutter change (0) Sometimes the shutters don't move fast enough. You can skip some scans after each shutter event if this is the case. -Ssize Set size of power spectra smoothing (1) -t[start,stop] Truncate scans (OFF) Truncate scans. Useful for files done in non-destructive read mode. Very similar to truncateir in CLASSIC reduction software. You can force a range by supplying numbers, or use the mouse if you do not supply two numbers. -u Toggle use dither freqs for fringes (ON) Use the dither frequencies for the fringe frequencies. When turned off the program will try and work the frequencies out from the power spectra by fitting Gaussian curves. -U[freq] Set DC suppression frequency (10.0 Hz) All frequencies below this are forced to zero power. -v Toggle verbose mode (OFF) Verbose mode prints out a lot of information about what is going on and is very helpful. -w[weight] Set minimum fringe weight (1.0) Minimum fringe weight to decide if a fringe is present or not. -W[width] Set PS peak width for fit (10.0 Hz) Default of width of power spectrum peak for fitting. See flag -p above. -zpixmult Set pixel multiplier for waterfall plot. The default is 2, but for large data sets you may wish to use 1, or for small ones a larger number.