|Not logged in|
|OMC Data Server - System Overview|
- OMC Data Server functionality
- Current status of the OMC Data Server
- OMC science analysis processing at CAB
- Column description of output FITS files (OMC light curves)
- PROBLEMS column description
- Known limitations
- Other foreseen functionality
The Optical Monitoring Camera (OMC) observes the optical emission from the prime targets of the gamma-ray instruments on-board the ESA mission INTEGRAL (launched on Oct. 17, 2002). OMC has also the capability to monitor serendipitously a large number of optically variable sources within its field of view.
Since only a number of OMC sub-windows (typically 100, and in any case less than 228) of 11x11 pixels can be downloaded to Earth, the targets to be monitored by OMC have to be pre-selected on ground. For this purpose, an OMC Input Catalogue were compiled containing:
- Most gamma-ray sources.
- Most X-ray sources.
- Most AGNs within the photometric limits of the OMC.
- Most variable stars (including eruptive variable stars, novae and cataclysmics).
- Several known additional optical variable objects.
- Hipparcos and Tycho reference stars for astrometric and photometric calibration.
The OMC Data Server includes all observations from revolutions 11-1879 (publicly available on Nov 28, 2018) and public observations up to revolution 2019. Data prior to revolution 951 are from revision 2 of the INTEGRAL Archive. From revolution 951 onwards revision 3 is being used. The Off-line Science Analysis software OSA 7.0 was used to process the OMC data until revolution 890. In revolution 891 the Off-line Science Analysis software was switched to OSA 9.0, which includes an improved PSF calculation with respect to OSA 7.0, and in revolution 1281 was switched to OSA 10.1 improving the centroiding capabilities in open loop slews and in some extreme cases (crowded fields, faint sources...). Concerning the OMC science analysis, OSA 10.1 is equivalent to the latest software (OSA 11.0). Work to update old data to revision 3 and their reprocessing with OSA 10.1 is ongoing.
The OMC Data Server includes completely the OMC Input Catalogue V0005, so the user can check if a given source has been pre-selected as a candidate to be monitored by the OMC. However the main aim of the OMC Data Server is to give a friendly interface to the OMC science results.
OMC Data Server functionality
The OMC catalogue comprises 541802 objects. The query to access the archive is made by means of an HTML fill-in form which permits to perform queries by object name, coordinates, object type, V-magnitude range, date of observation, time binning, centroid method and/or number of points of the light curve. The output data may be ordered by object name, coordinates, magnitude or date and time of observation. Two output formats are available: HTML or ASCII.
The system has a built-in name resolver utility which makes possible to query the archive using any of the object names provided by SIMBAD. The name resolver gives more than three million and a half identifications for the astronomical objects contained in the OMC catalogue. The full list of the names associated to a given object can be obtained by simply clicking on the target name in the output form.
Results from Search
The following utilities are provided in HTML output format to the users with proper access rights:
- Plot utility: A browse plot of a light curve can be generated on-the-fly by clicking on the corresponding link.
- Fits Header Display: Links are provided to display the FITS headers of each requested light curve file.
- Data Retrieval: Light curves may be retrieved individually or in groups. If a single light curve is requested, it is delivered as an uncompressed FITS file. Multiple light curve retrieval generates a packed file in zip format.
- Help-Desk: A Help Desk facility to channel questions and to provide continuous support to users of the archive is provided.
Current status of the OMC Data Server
The access to the OMC Input Catalogue data is open for everyone. It allows to know which objects have been monitored by the OMC, and when. It also provides a cross-correlation window based on the SIMBAD Database.
Security and privacy of the private data are assured in two ways: user authentication (username+password) and encrypted data transfer. There are several types of user profiles, each of them with different data access policies. The restricted interface to access PV phase and Core Programme data is open only for the INTEGRAL teams. To obtain your password please send an e-mail to email@example.com.
Currently, you can retrieve from the OMC Data Server the processed light curves. To get the original CCD windows for a manual reprocessing you have to use the standard interface at ISDC. We plan to give also this functionality in the future.
OMC Data Server statistics
Total number of observed sources (at least one photometric point): 243050 Number of "scientific" observed sources (at least one photometric point): 158795 Number of scientific sources with more than P photometric points (time binning of 10 minutes):
Num. points (P) Num. of sources > 50 91347 > 100 66395 > 200 43199 > 500 18981 > 1000 8133 > 2000 2676 > 3000 1081 > 4000 395 > 5000 184
Those sources with more than 50 photometric points (time binning of 10 minutes) and with object type available in the Simbad Database, show the following classification:
Object type Number of sources Variable Star 7022 Variable Star of Mira Cet type 2912 Variable Star of RR Lyr type 2653 Eclipsing binary of Algol type 1896 Semi-regular pulsating Star 1424 Quasar 1275 Pulsating variable Star 1209 Pulsars 669 Possible Quasar 645 Eclipsing binary 520 Radio Galaxy 442 Emission-line galaxy 400 T Tau-type Star 389 Classical Cepheid (delta Cep type) 378 Eclipsing binary of beta Lyr type 365 X-ray source 306 Eclipsing binary of W UMa type 270 Nova 261 gamma-ray source 256 Flare Star 238 Galaxy 236 Classical Cepheid variable Star 233 Emission-line Star 229 Variable Star of Orion Type 228 Dwarf Nova 202 Seyfert 1 Galaxy 181 Low Mass X-ray Binary 173 Seyfert 2 Galaxy 171 Carbon Star 165 Variable Star of irregular type 144 High Mass X-ray Binary 117 Emission Object 112 Infra-Red source 103 BL Lac - type object 95 Variable Star with rapid variations 94 Active Galaxy Nucleus 85 Star in Cluster 79 Variable Star of RV Tau type 60 Seyfert Galaxy 50 Star 50 Variable Star of W Vir type 46 Radio-source 43 Be Star 42 Variable Star of alpha2 CVn type 42 Nova-like Star 42 Variable Star of beta Cep type 41 Spectroscopic binary 36 Star suspected of Variability 35 Variable Star of delta Sct type 35 HII Galaxy 35 Cataclysmic Variable Star 32 SuperNova Remnant 30 Symbiotic Star 30 Planetary Nebula 29 Variable of RS CVn type 26 Eruptive variable Star 20 Wolf-Rayet Star 19 Cataclysmic Var. AM Her type 18 LINER-type Active Galaxy Nucleus 17 Cluster of Galaxies 17 Variable of BY Dra type 16 X-ray Binary 14 Blue object 13 S Star 13 Variable Star of R CrB type 12 UV-emission source 12 White Dwarf 11 Variable White Dwarf of ZZ Cet type 10 Galaxy in Pair of Galaxies 10 HII (ionized) region 8 Galaxy in Cluster of Galaxies 8 Pair of Galaxies 8 Elliptical variable Star 7 Cataclysmic Var. DQ Her type 7 Rotationally variable Star 7 Star in double system 7 Double or multiple star 6 Possible Planetary Nebula 6 High proper-motion Star 5 gamma-ray Burster 5 Gravitationnaly Lensed Image of a Quasar 5 Low Surface Brightness Galaxy 5 Variable Star of FU Ori type 4 Object of unknown nature 4 SuperNova Remnant Candidate 4 Group of Galaxies 3 Star in Nebula 3 SuperNova 3 Nebula of unknown nature 3 Blazar 2 Cluster of Stars 2 Maser 2 Starburst Galaxy 2 Globular Cluster 2 Star in Association 2 Young Stellar Object 1 Molecular Cloud 1 Blue compact Galaxy 1 Interacting Galaxies 1
OMC science analysis processing at CAB
Around 100 OMC CCD sub-windows, each of 11x11 pixels, are downloaded to Earth about every minute, without any on-board processing. The data are received at the Integral Science Data Center, where they are completely processed in an automatic way. OMC data are then re-processed at CAB and stored in the OMC Data Server. The principal steps of this science analysis processing are the following:
The information in the telemetry packages is converted to CCD pixel values.
The OMC CCD sub-windows are corrected from bias and dark current. They are then convolved with the corresponding flat-field matrix.
OMC CCD sub-windows obtained within periods of around 10 minutes (default value) as well as within the full Science Window are combined to obtain a better signal-to-noise ratio. A Science Window is a pointing (typically around 30 minutes) or a part of it in case of being too long (staring mode). A third solution without any combination of the OMC CCD sub-windows is also given. These three cases correspond to three different processing options.
The photometric value for each source of interest is obtained by applying three different extraction masks of 1x1, 3x3 (default value) and 5x5 pixels to the combined images (see column description below). The local background, as measured on a corona around the extraction mask, is subtracted. If Source coordinates (default) is selected as the centroid method in the fill-in form, the position of the extraction mask is re-centred on the source coordinates given in the OMC Input Catalogue, allowing a maximum offset of 10 arcsec (parameter IMA_maxWcsOff in OMC OSA). Instead if Brightest pixel is selected as the centroid method, the position of the extraction mask is re-centered on the brightest pixel within the central 5x5 pixel section. This extraction software can also be run offline to modify the parameters, in order to minimize the effects of background stars when needed.
The photometric values are stored as V magnitudes, thus compiling light curves with three time binnings:
- Shot-by-Shot: one photometric point per shot.
- 10 minutes: around one photometric point each 10 minutes. Shots with exposure smaller than 20 seconds are rejected to increase the signal-to-noise ratio.
- ScW-by-ScW: one photometric point per Science Window. Shots with exposure smaller than 60 seconds are rejected to increase the signal-to-noise ratio.
We must notice that due to the rejection of the shortest shots when using time binnings of 10 minutes and ScW-by-ScW, the brightest sources will not be processed when selecting these time binnings, because they will be saturated in the remaining shots. For these sources, the user should select the time binning of Shot-by-Shot, and then filter the results to keep only the shortest shots with no saturation effects.
In this science analysis, OMC Input Catalogue Version V0005 and the following versions of OMC OSA 7.0, OSA 9.0 and OSA 10.1 (Off-line Science Analysis) components were used:OSA 7.0 (revolutions 11-890): omc_science_analysis 5.3.1 omc_scw_analysis 5.3.1 o_cor_science 5.3.1 o_cor_box_fluxes 6.5.1 o_gti 5.3.1 gti_create 2.2 gti_attitude 1.5 gti_merge 1.6 o_src_analysis 5.3.1 o_src_get_fluxes 8.1 o_src_compute_mag 4.4 omc_obs_analysis 5.3.1 o_src_collect 2.4 OSA 9.0 (revolutions 891-1280): omc_science_analysis 6.0 omc_scw_analysis 6.0 o_cor_science 6.0 o_cor_box_fluxes 6.5.1 o_gti 6.0 gti_create 2.3 gti_attitude 1.5 gti_merge 1.6 o_src_analysis 6.0 o_src_get_fluxes 8.2 o_src_compute_mag 4.4 omc_obs_analysis 6.0 o_src_collect 2.4 OSA 10.1 (revolutions 1281-->): omc_science_analysis 6.0.2 omc_scw_analysis 6.0.2 o_cor_science 6.0.2 o_cor_box_fluxes 6.5.1 o_gti 6.0.2 gti_create 2.3 gti_attitude 1.5 gti_merge 1.6 o_src_analysis 6.0.2 o_src_get_fluxes 8.3 o_src_compute_mag 4.4 omc_obs_analysis 6.0.2 o_src_collect 2.4
Best photometric accuracy achievable
Expected best photometric accuracy in V magnitudes of the OMC for different V values and various effective integration times. A typical observation totals 300 s of effective integration time.
V magnitude Eff. integration (s) 8 10 12 14 16 10 0.007 0.02 0.1 -- -- 300 -- 0.005 0.01 0.04 0.3 900 -- 0.003 0.006 0.026 0.17
The above values do not account for possible systematic biases.
In August 2006 a new set of flatfield matrices (Version 5) as well as photometric calibration files (Version 5) were delivered by the OMC team to ISDC. These new IC (Instrument Characteristic) files improved substantially the old ones (Version 4).
The data currently available at the OMC Data Server were analyzed using the new IC files. In the following table the version number of each IC file used in the analysis is given:
IC Data Structure Version OMC.-PHOT-CAL 5 OMC.-FLAT-CAL 5 OMC.-BDPX-CAL 2 OMC.-DARK-CAL 2 OMC.-GOOD-LIM 7 INTL-GOOD-LIM 2
Column description of output fits files (OMC light curves)
REVOL Revolution number valid for time of data taking SWID Science Window identifier from which this row was taken TFIRST Time of the first data element in ISDC Julian days BARYTIME Barycentric time for the first data element in ISDC Julian days TELAPSE Elapsed time of the integration in seconds EXPOSURE Effective integration time in seconds SHOTTYPE Type of shots used for building integration (1=Photometric, 2=Science) OMC_ID OMC catalogue source identifier (IOMC number) TYPE_TAR Target type (1=Photometric, 2=Science, 3=Dark Current) RA_OBJ Source right ascension in degrees DEC_OBJ Source declination in degrees FLUX_1 Flux in electron/s derived from 1x1 integration boxes ERFLUX_1 Error estimate for FLUX_1 FLUX_3 Flux in electron/s derived from 3x3 integration boxes ERFLUX_3 Error estimate for FLUX_3 FLUX_5 Flux in electron/s derived from 5x5 integration boxes ERFLUX_5 Error estimate for FLUX_5 SKYBACK Mean flux from sky background in electron/pixel/s SKYERROR Error estimate for SKYBACK SIZE_MAG Integration box size for deriving MAG_V MAG_V Computed V (Johnson) magnitude (default value) ERRMAG_V Error estimate for V magnitude CATMAG_V Catalog V (Johnson) magnitude CATERR_V Catalog error estimate for V magnitude MAG_V1 Computed V magnitude for the 1x1 pixel area ERMAG_V1 Error estimate for V magnitude in 1x1 pixel area MAG_V3 Computed V magnitude for the 3x3 pixel area ERMAG_V3 Error estimate for V magnitude in 3x3 pixel area MAG_V5 Computed V magnitude for the 5x5 pixel area ERMAG_V5 Error estimate for V magnitude in 5x5 pixel area PROBLEMS Flag for various problems (0 = no problems) NOISE_LL Read-out noise in e- (low gain, left ROP) NOISE_LR Read-out noise in e- (low gain, right ROP) NOISE_HL Read-out noise in e- (high gain, left ROP) NOISE_HR Read-out noise in e- (high gain, right ROP) CENTRING_X Derived X-axis offset of the source from the box centre (pixels) CENTRING_Y Derived Y-axis offset of the source from the box centre (pixels) PSF_FWHM Effective PSF FWHM in pixels X_TAR X coordinate of the lower left pixel of the target box (pixels) Y_TAR Y coordinate of the lower left pixel of the target box (pixels) RANK On-board sequence number of box RA_FIN Derived right ascension in degrees RA_FIN_ERR Standard error for RA_FIN*cos(DEC_FIN) DEC_FIN Derived declination in degrees DEC_FIN_ERR Standard error for DEC_FIN
PROBLEMS column description
This is the meaning of PROBLEMS column in output fits files. Problems are stored in an unsigned integer register, and any problem encountered is logically ANDed to the existing register value. Deconstruction of the total into its only possible component values reveal the individual PROBLEMS. Problem values 64 and 2048 are reserved for future improvements.
NAME VALUE DESCRIPTION OMC_PROBLEM_NONE 0 No problems OMC_PROBLEM_CENTROID_OUT 1 Centroid does not match maxWcsOff radius; The WCS position has been used as centroid OMC_PROBLEM_EXTRAPOLATED_MAG 2 The magnitude was extrapolated OMC_PROBLEM_BAD_CENTROID 4 No centroid is available or is inaccurate OMC_PROBLEM_BAD_PSF 8 Bad PSF. A default value was used OMC_PROBLEM_ANOMALOUS_PSF 16 The PSF shape is anomalous OMC_PROBLEM_LOW_FLUX_1 32 Flux of central pixel too low OMC_PROBLEM_BADPIXEL_SKY 128 Bad pixel found in sky background OMC_PROBLEM_BADPIXEL_RIM_5 256 Bad pixel found in 5x5 rim OMC_PROBLEM_BADPIXEL_RIM_3 512 Bad pixel found in 3x3 rim OMC_PROBLEM_BADPIXEL_RIM_1 1024 Central pixel bad OMC_PROBLEM_SKY_ERROR 4096 Sky error larger than accepted limit OMC_PROBLEM_UNKNOWN_MAG 8192 Magnitude could not be calculated OMC_PROBLEM_EXTND_SRC 16384 Source is extended - flux not valid OMC_PROBLEM_COORD_OUT 32768 WCS position close to the edge or out of the OMC box; The brightest pixel has been used.
The automatic extraction of fluxes and magnitudes produces reliable results only for point like sources when compared with the OMC pixel size.
If the source coordinates are inaccurate by more than 2 OMC pixels (around 35 arcsec), the software analysis will not be able to re-centre the target and the derived fluxes and magnitudes obtained with default analysis parameters will not be correct.
For extended sources or high energy source counterparts with large uncertainties in their position, the OMC planning assigns multiple adjacent sub-windows to cover the whole area. In that case, multiple boxes are found with different ranks but with the same OMC_ID. From OSA 6.0 onwards these adjacent sub-windows can be correctly analyzed by using the Source coordinates option as centroid method (default in the fill-in form). In this case a virtual 11x11 pixel sub-window is created inside the whole area centred at the source position. After that, OSA works on this new sub-window and ignores the previous sub-windows mosaic. This is an internal software trick, these virtual sub-windows do not exist as standard sub-windows. Note that if Brightest pixel is selected as the centroid method, these adjacent sub-windows will not be analyzed correctly as the software treats each box individually.
If another star is within a few pixels of the source of interest, it can introduce systematic errors in the derived fluxes and magnitudes. The strength of this effect can be different for different pointings, since the relative position in the sub-windows will slightly change for different rotation angles.
Since OSA 4.0 the detection of saturated sources has been improved considerably. However, some of the bright sources slightly saturating one or few pixels might not be detected as saturated sources. As a consequence, their derived magnitudes are not correctly computed. The observer should check whether the source might be saturating the CCD for a given integration time, and reanalyse the data rejecting the shots with the longest integration times.
Due to thermoelastic deformations, the alignment of the OMC optical axis with the S/C attitude reference (after correcting the OMC misalignment) may diverge by up to 30 arcsec (around 2 OMC pix). From OSA 5.0 onwards, the derived coordinates are corrected at the time of computing the WCS support by using the photometric reference stars, giving an accuracy better than 2 arcsec in most cases.
Other foreseen functionalities
A variety of data analysis tools are being developed and will be available in the near future. The aim of these tools is to provide added-value functionalities to the system giving the ability to perform data analysis tasks remotely.
Retrieval of OMC images: The user will be able to get raw and corrected images. In the last case the OMC sub-windows will be corrected for bias, dark current and flatfield.
Time series analysis: The operation of INTEGRAL from a high orbit allows a continuous observation (only interrupted by the radiation belts crossing) for periods of several weeks giving a unique photometric capability that cannot be addressed from ground-based observatories. Our aim is that the OMC data server also provides information in the frequency domain. Given the diversity of the OMC targets (AGNs, X-rays binaries, gamma-ray burst, eclipsing binaries, pulsating variables, ...) and the variability patterns (long/short-term variations, mono/multi-periodicity,...), a detailed study on the techniques to be applied in each case must be performed.
Data mining. Light curve characterization: Even though the OMC data server allows making queries by object type based on the classification provided by SIMBAD, it is clear that this classification can be greatly improved with the use of the OMC data. Given the vast amount of data to be handled, classification procedures based on the visual inspection by experts are not adequate and data mining techniques must be used instead. We are presently working in a neural network system to classify light curves of periodic variable stars. The network will be trained with the HIPPARCOS light curves and, in a first step, it will allow for the automatic classification of eclipsing binary stars with a further extension to other types of variable stars (e.g. pulsating variables). The system is designed to perform a unsupervised topological mapping based on morphological proximity among the light curves.
Version 3.0 - March 2012 © CAB (INTA-CSIC) Home