PyLMT: a transient detection pipeline for the 4-m International Liquid Mirror Telescope Open Access

ABSTRACT

The International Liquid Mirror Telescope (ILMT) is a 4-m aperture, zenith-pointing telescope with a field of view of 22|$^{\prime }$|⁠, situated in the foothills of the Himalayas. The telescope operates in continuous survey mode, making it a useful instrument for time-domain astronomy, particularly for detecting transients, variable stars, active galactic nuclei variability, and asteroids. This paper presents thePyLMT transient detection pipeline to detect such transient/varying sources in the ILMT images. The pipeline utilizes the image subtraction technique to compare a pair of images from the same field, identifying such sources in subtracted images with the help of convolutional neural network (CNN)-based real/bogus classifiers. The test accuracies determined for the real/bogus classifiers ranged from 94 per cent to 98 per cent. The resulting precision of the pipeline calculated over candidate alerts in the ILMT frames is 0.91. It also houses a CNN-aided transient candidate classifier that classifies the transient/variable candidates based on host morphology. The test accuracy of the candidate classifier is 98.6 per cent. It has the provision to identify catalogued asteroids and other Solar system objects using public data bases. The median execution time of the pipeline is approximately 29 min per image of 17 min exposure. Relevant CNNs have been trained on data acquired with the ILMT during the cycle of 2022 October–November. Subsequent tests on those images have confirmed the detection of numerous catalogued asteroids, variable stars, and other uncatalogued sources. The pipeline has been operational and has detected 12 extragalactic transients, including 2 new discoveries in the 2023 November–2024 May observation cycle.

1 INTRODUCTION

Modern sky survey programmes have enabled the discovery of transient events in large numbers. Advances in imaging technologies, like the charged coupled device (CCD), along with enhanced computational facilities, have been the key drivers behind the facilities conducting these programmes. Furthermore, such surveys have significantly expanded the scope of astronomical research by enabling systematic and continuous monitoring of large portions of the sky. These surveys provide a comprehensive view of the dynamic universe, allowing for the study of a wide variety of celestial objects, from transients occurring in distant galaxies – such as supernovae, tidal disruption events, and active galactic nuclei variability – to stellar variability in nearby stars, including pulsating variables and eclipsing binaries. Additionally, they play a crucial role in the detection and monitoring of Solar system objects like asteroids and comets (Denneau et al. 2013; Ye et al. 2019; Duev et al. 2021). The ability to capture both rare and common phenomena with such surveys has significantly enhanced our understanding of the life cycle of stars, mechanisms driving cosmic explosion and origins of the building blocks of the Solar system.

The Palomar Transient Factory (PTF; Law et al. 2009) was a fully automated, wide-field survey dedicated to systematic exploration of the optical transient sky. The survey utilized an 8.1 deg|$^{2}$| CCD camera mounted on the 48-inch Samuel Oschin telescope at Palomar Observatory to perform imaging. The colours and light curves of the detected transients were obtained using the Palomar 60-inch telescope. The Zwicky Transient Facility (ZTF; Bellm et al. 2019) is a new optical transient survey facility that is the successor to the PTF. It uses the refurbished Palomar 48-inch telescope equipped with a custom-build wide-field CCD camera with a field of view (FoV) of 47 deg|$^{2}$|⁠. The facility yields more than an order of magnitude improvement in survey speed over its predecessor. As a result, it has discovered a plethora of transients over the years. Other survey programmes that provide transient alerts include the Panoramic Survey Telescope and Rapid Response System (Pan-STARRS; Chambers et al. 2016), Gaia (Gaia Collaboration 2016), Asteroid Terrestrial-impact Last Alert System (ATLAS; Tonry et al. 2018), and Gravitational-wave Optical Transient Observer (GOTO; Steeghs et al. 2022).

Such surveys conduct repeated imaging of the sky, utilizing real-time image subtraction to detect new transient events (Cao, Nugent & Kasliwal 2016) and provide alerts for follow-up observations. Large volumes of image data acquired through these surveys in a single night make manual methods of finding transients using this method prohibitive. Hence, several automated techniques for detecting transient candidates using image subtraction have been explored and implemented recently (Bailey et al. 2007). Techniques, like boosted decision trees and random forest classifiers, have been successfully implemented in previous surveys like Nearby supernova factory (Wood-Vasey et al. 2004), PTF (Brink et al. 2013), and intermediate PTF (iPTF; Bellm, Kulkarni & Intermediate Palomar Transient Factory Collaboration 2017). Convolutional neural network (CNN; Lecun et al. 1998) based techniques (Andreoni et al. 2017; Cabrera-Vives et al. 2017; Gieseke et al. 2017; Duev et al. 2019; Mahabal et al. 2019; Turpin et al. 2020; Makhlouf et al. 2022; Acero-Cuellar et al. 2023; Santos et al. 2024) for transient detection have been introduced in recent programmes like the ZTF. Transient identification using shapelet analysis has been explored in Ackley et al. (2019). Further categorization of the candidates into subcategories like SNe, variable stars, asteroids, active galactic nuclei (AGNs), and artefacts based on time-series and context data have been implemented for ZTF candidate alerts (Carrasco-Davis et al. 2021; Sánchez-Sáez et al. 2021). The candidates are confirmed/rejected separately using photometric and spectroscopic follow-up observations.

Another such survey, the 4-m International Liquid Mirror Telescope (ILMT; Surdej et al. 2006; Borra, Hickson & Surdej 2009; Surdej et al. 2018; Kumar et al. 2022a) has recently begun operations in Devasthal, India. It is the first optical survey telescope in India which performs repeated imaging of the zenith sky with a cadence of 1 d, making it an instrument capable of detecting new and interesting transients (Kumar et al. 2018). As a zenith telescope, it offers several advantages, including low observational air mass, optimal image quality, minimal light pollution, and no time lost to pointing adjustments. This capability of the ILMT motivated the development of a transient detection pipeline called the PyLMT. The objectives of the pipeline are twofold (i) use image subtraction to detect transient candidates automatically and in real-time, and (ii) categorize the detected candidates into distinct classes, streamlining the identification and retrieval of target candidates during the manual vetting process of the pipeline outputs. Additionally, the pipeline makes use of the catalogues namely Gaia (Gaia Collaboration 2016), the Set of Identifications, Measurements and Bibliography for Astronomical Data (SIMBAD; Egret, Wenger & Dubois 1991), and Institute of Celestial Mechanics and Computation of Ephemerides (IMCCE; Berthier et al. 2006) to confirm or reject candidates based on the type of underlying object (e.g. bright star, known galaxy, known minor planet). The pipeline also uses a context-aware adaptive detection approach to reduce false positive detections. A rigorous discussion of all the steps in the pipeline, relevant design parameters, module-wise tests, pipeline implementation, and resulting transient detections and discoveries is presented in this paper.

The paper is structured as follows. Section 2 covers the basic structural and functional aspects of the ILMT. The overall description of the pipeline, encompassing the image subtraction module and the transient detection and candidate classification steps, is detailed in Section 3. Various tests made on the pipeline and corresponding results, along with implementation on ILMT science images, are given in Section 4. Finally, conclusions and discussions about the prospects of the pipeline are presented in Section 5.

2 THE 4-M INTERNATIONAL LIQUID MIRROR TELESCOPE

The ILMT achieved its first light in April 2022. It is a 4-m aperture f/2.36 zenith-pointing telescope with a rotating bowl filled with |$\sim$|50 L of mercury constituting the primary reflecting surface. The balance between the centrifugal force and the force of gravity maintains the paraboloid shape of the mirror. The bowl is a carbon fibre-epoxy skin over a closed foam core and is mounted over an air-bearing, supported by a 3-point mount system (Surdej et al. 2006; Borra et al. 2009; Surdej et al. 2018; Kumar et al. 2022a). The mirror is surrounded by a metallic structure that houses the optical corrector and a 4 K × 4 K CCD detector. The telescope’s effective FoV is 22 arcmin. A relatively large aperture enables the detection of faint sources of magnitudes up to 21.9 in Sloan Digital Sky Survey (SDSS) g|$^{\prime }$| band (Ailawadhi et al. 2024). The telescope structure is shown in Fig. 1.

Due to the fixed-pointing nature of the telescope, the time-delay integration (TDI) technique is employed to perform imaging. The technique compensates for the sidereal motion of the sky to give stationary images. The CCD remains fixed and is read out at the exact sidereal rate along the east–west direction. The continuous read-out ensures propagation of accumulated charge at the same sidereal rate, thereby counterbalancing the star trails and rendering point-like stellar images. This technique was first implemented with the Steward Observatory 1.8-m CCD/Transit Instrument (CTI; McGraw et al. 1982). It has been observed that the TDI imaging suffers from a north–south elongation at non-zero latitudes due to conical trajectories of stellar images on the CCD plane (Gibson & Hickson 1992). The optical corrector affixed in front of the focal plane ensures that the conical trajectory traversed by the sources on the CCD plane is made rectilinear (Hickson & Richardson 1998; Negi et al. 2024). The parameters of the ILMT are given in Table 1.

The ILMT covers the same local sidereal time (LST) fields on successive nights. The entire science image covers 1.21 deg|$^2$| of the sky and is acquired during a 17-min exposure in TDI mode. Images are acquired in a single filter for any given night. Depending on the season, up to 35 science frames can be acquired on a typical observation night with favourable weather conditions. These images are processed using the PyLMT pipeline to search for astronomical transients, asteroids, and other variable sources. Up to now, three observation cycles have concluded: 2022 October–November, 2023 March–June, and 2023 November–2024 May.

3 PIPELINE DESCRIPTION

The pipeline consists of three modules, namely ILMTDiff, TransiSearch, and NovaNet for performing image subtraction, transient detection, and transient candidate classification, respectively. A concise overview of the three basic steps is also provided in Pranshu et al. (2024a). Fig. 2 illustrates the overall architecture of the PyLMT pipeline. Before ingestion to the pipeline, the images have to be corrected for dark, flat, sky background, and astrometry (Kumar et al. 2022b; Dukiya et al. 2024; Negi et al. 2024). A series of python and bash scripts coordinate the various pre-processing and transient detection steps applied to the acquired ILMT images and across multiple workstations.

The ILMT frames are acquired as long ‘strips’ with dimensions of 22 arcmin × 220 arcmin, where the initial 22 arcmin corresponds to the TDI ramping region and must be discarded. The TDI ramping occurs because not all sources traverse the full length of the CCD at the start of image acquisition. The east–west orientation of the CCD ensures the orientation of the frames along the RA (refer to Fig. 3). This results in the frames having unique RA coverage determined by the starting LST and the TDI exposure time (fixed at 17 min).

For every acquired (science) image, a reference image with the nearest acquisition LST is selected from a data base. The images are registered using either the WCS information in the FITS header or using the astroalignpython software (Beroiz, Cabral & Sanchez 2020). An efficient implementation of the image subtraction technique requires a minimal spatial variation of the point spread function (PSF). Therefore, the pre-processed ILMT frames are segmented into 1051|$\times$|1051 pixel sections before subsequent steps. The adjacent sections overlap by 26 pixels to prevent potential detection failures due to ‘blind zones’ between segments.

3.1 Image subtraction

Images from survey telescopes like the ILMT are typically populated with numerous astronomical sources. The sources exhibiting non-varying flux are removed to facilitate the isolation of transient and variable sources. This is accomplished using the image subtraction technique wherein a reference image – representing the same region of the sky – is subtracted from the science image to eliminate non-varying sources. The ILMTDiff module performs the image subtraction in the pipeline.

The science and reference images are first registered and then transferred to this module. The two important steps performed within the module are flux scaling and PSF matching. The science and reference images are acquired on two different nights with possibly different visibility and seeing conditions. This results in image pairs having different source full width at half-maximum (FWHM) and flux counts. Image subtraction requires these two parameters for both images (cut-outs in our case) to be matched. Scaling for flux counts is performed by multiplying the reference cut-out image with a factor equal to the ratio of source fluxes in science and reference images. The reference image is then convolved with an optimal convolution kernel to match the PSFs. This ideally requires the reference image to be acquired in better-seeing conditions (hence smaller PSF FWHM) than the science image as convolution degrades the PSF (Huckvale, Kerins & Sale 2014). Finally, the science and convolved reference images are subtracted to produce a difference image. Fig. 4 illustrates the result of image subtraction performed on a 1051 × 1051 pixel cut-out of an ILMT image.

This work builds upon Bramich (2008), where the convolution kernel is represented as a square matrix with pixel entries being the parameters to be optimized. Image subtraction is transformed into an optimization problem, with the squared sum of subtraction residuals being the objective function to be minimized. Residuals are evaluated for pairs of common individual sources in science and reference cut-outs. A scipy (Virtanen et al. 2020) hosted numerical least-squares optimizer is used to determine the optimal kernel. Detailed discussion on the ILMTDiff module is given in Appendix  A.

3.2 Transient detection

Detecting transients in subtracted images requires filtering of the artefacts that can obscure genuine detection. Common sources of these artefacts include astrometric misalignment, improper subtraction, cosmic hits, random background fluctuation, and defective CCD columns. They are typically characterized by their ‘non-PSF-like’ morphology (refer to Fig. 5), which deviates from the expected profile of ‘real’ astrophysical sources. A two-step strategy was adopted for the TransiSearch transient detection module to remove the artefacts. In the first step, the CNN-based ‘real/bogus’ classifier is implemented to reject most artefacts (a.k.a ‘bogus’ sources). Subsequently, threshold-based filtering is applied to the classifier output to enhance the purity of the candidates. Table 2 lists the source parameters and respective thresholds used in the module.

A training data set of nearly 5000 samples was created to train the CNN classifier. The ‘real’ training set consisted of ‘PSF-like’ real astrophysical sources (Fig. 6) extracted from the ILMT frames. The ‘bogus’ training set consisted of visually inspected artefact sources extracted from the subtracted images. Every sample in the data set has a dimension of 31 |$\times$| 31 pixels and is the same as the input dimension of the CNN model. To reduce false positive detections, a two-CNN-based adaptive detection strategy is used (discussed in detail in Section 3.5).

The TransiSearch module extracts all the sources above 4|$\sigma$| significance in the subtracted image using find_peaks algorithm of the photutilspython software (Bradley et al. 2016). The sources are extracted as square cut-outs of the dimension 31 |$\times$| 31 pixels. They are subsequently pre-processed and classified using the trained CNN model discussed before. The CNN assigns a score to each of the detected sources. A classification threshold of 0.5 is applied to these scores to classify respective sources as ‘real’ or ‘bogus’. Data set preparation and training of the ‘real/bogus’ classifier is discussed in detail in Appendix  B.

3.3 Transient candidate classification

Multiple transient candidates are generated in the previous step for every ILMT frame. Real-time classification of the candidates is pivotal to planning the most appropriate follow-up observations. In particular, the early-time photometric and spectroscopic follow-up of SNe can be advantageous in obtaining constraints on its physical properties (Khazov et al. 2016). The NovaNet module in the pipeline classifies the TransiSearch candidates into three categories namely ‘extended-host’, ‘point-host’, and ‘hostless’. Following the classification, the candidates undergo cross-matching with existing catalogues (discussed in Section 3.4), thereby streamlining their selection process for follow-up observations.

The classification is based on the morphology of the object (if present) in the reference image around the position of detection. The ‘extended-host’ candidates are characterized by an associated extended galaxy (e.g. SN with an extended host galaxy, AGN), ‘point-host’ by a star/point-like source (e.g. variable star and QSO), and ‘hostless’ by the absence of source in the reference image (e.g. asteroids, CVs, SN with faint/invisible host). The ‘extended-host’ candidates, due to their potential association with bright SNe, are prioritized as triggers for follow-up observations.

The module uses a combination of trained CNN-based transient candidate classifiers to perform candidate classifications. The training data set for the CNNs was compiled with around 1600 cut-out samples curated with 102|$\times$|102 pixel image cut-outs of galaxies, stars, and empty spaces (Figs 7–9). Data augmentation techniques were employed to enhance the strength and diversity of the data set. Additionally, a separate data set with a cut-out size of 31 |$\times$| 31 pixel was compiled to train another transient candidate classifier for that specific input size. This separate data set was constructed by cropping the central 31|$\times$|31 pixel portion from 102|$\times$|102 pixel samples in the original classifier data set. The training data sets for candidate classifiers are summarized in Table 3. When a source is detected at any position in the difference image, a 102|$\times$|102 pixel cut-out and a 31|$\times$|31 pixel cut-out are extracted from the reference image around that position. The two cut-outs are then pre-processed and passed through an ensemble of multiple CNNs for classification. The final classification of the candidate is determined based on CNN scores associated with these two cut-outs. A detailed discussion on data set preparation, training, and logical organization of the CNNs is given in Appendix  C.

3.4 Catalogue cross-matching

Early-time identification of transient candidates like SNe is the primary goal of the PyLMT pipeline. After the majority of the artefacts have been removed in the previous steps, astrophysical sources like asteroids and variable stars turn out to be the major contaminants that should be removed. To that end, a catalogue cross-matching step was integrated into the pipeline. This additional step generates two distinct sets of candidates for each frame: (1) A comprehensive list consisting of all the detected transient and variable sources, and (2) a filtered list consisting of candidates after rejecting catalogued asteroids and probable variable stars.

The cross-matching step is hosted within the NovaNet module. The query to respective catalogues is performed using astroquery (Ginsburg et al. 2019) in python. The SkyBot service (Berthier et al. 2006) of the IMCCE is queried for all the candidates to reject Solar-system objects within 10 arcsec from their detection position. Subsequently, the Gaia G-band magnitude is queried for the remaining ‘point-host’ candidates within 1 arcsec of the detection position. Such candidates with a corresponding magnitude brighter than 19 are rejected for being probable variable stars. The magnitude threshold is set to ensure that any distant SNe candidate, whose host galaxy might appear as a faint point source, is not erroneously rejected.

The SIMBAD catalogue is queried to identify any known object (e.g. a catalogued galaxy, star, and QSO) underlying the detected candidates within 10 arcsec from the position of detection. For every matched counterpart, the object name and type are recorded under host-name entry in the transient candidate alert (refer to Section 3.6). This is particularly useful in indicating the detection of variable stars, QSOs, CVs, or SNe with established host galaxies (see Figs 17 and 18). Candidates associated with a catalogued galaxy are retained in the filtered outputs; for other cases, the SIMBAD cross-matching step serves as a supplementary feature without altering the final classification status. Extended-host candidates that do not correspond to known Solar-system objects are prioritized for follow-up observations and reported to the transient name server (TNS). The flowchart representation of the entire catalogue cross-matching step is shown in Fig. 10.

3.5 Adaptive detection

‘Precision’ and ‘recall’ are two important metrics used to evaluate the performance of binary classifiers like the ‘real/bogus’ classifier. They can be expressed mathematically using the following relations:

A model optimized for high recall is capable of detecting fainter transients with a lower signal-to-noise (S/N) ratio, simultaneously being more susceptible to false positive outputs. Alternatively, a model optimized for higher precision exhibits a reduced false positive rate but at the cost of an increased fraction of missed detections, especially at fainter magnitude limits. A solution consisting of context-aware implementation of both varieties of models was adopted to minimize false positives while maintaining acceptable levels of detection sensitivity.

Upon performing a simple inspection of the candidate output, it was established that the ‘point-host’ candidates (primarily due to improper subtraction of non-variable stars) represented the majority of false positive detections. The adopted solution involves a pre-emptive classification of all sources (both real and bogus) in the difference image into ‘point-host’ and ‘non-point-host’ categories using CNN. Sources classified as ‘point-host’ by the CNN are forwarded to a high-precision classifier and the rest to a high-recall classifier. This ensures that ‘point-host’ sources, which previously contributed to a significant number of false positives, undergo more rigorous scrutiny compared to other sources. This provision prioritizes the detection of fainter, high-priority ‘non-point-host’ transients (e.g. faint SNe), while extremely faint ‘point-host’ transients or variable stars may remain undetected. This provision was implemented within the TransiSearch module itself.

The original ‘real/bogus’ classifier can be used as the high-recall classifier while a separate classifier was needed for the high-precision classifier. One possible method to create such a classifier was to take a high-recall CNN model and increase the classification threshold. Another method was to bias the training data set towards artefact samples. This makes the trained model better at identifying and rejecting artefacts, thereby reducing false positives. The second method was adopted for the NovaNet module. The implementation of this technique resulted in a reduction of false positive detections in final output from around 35 per cent to around 10 per cent. The training data sets for high-precision and high-recall classifiers are summarized in Table 4.

3.6 Command line implementation and candidate alert interpretation

The PyLMT is readily executable using linux command line interface (CLI) from inside the science image directory. A list of tunable parameters of the pipeline is listed in Table C1. The path to the reference image directory has to be specified with the command. The appropriate reference image is selected for a science image using the astrometric information stored in the fits header. The result for transient detection and classification (referred to as alerts) is stored in a zip folder containing relevant PDF and CSV files.

Each candidate alert contains six entries (shown in Figs 17 and 18), namely Solar system object that indicates whether it is a minor planet (if yes then the corresponding MPC name and V-band magnitude is printed), confidence score that is an ordered pair of CNN confidence scores of classification and detection, respectively, X–Y coordinates and WCS coordinates of candidate, host type that indicates the corresponding NovaNet classification and host name that indicates the source name retrieved from SIMBAD corresponding to the candidate (e.g. a catalogued host galaxy, variable star, and QSO). The date and starting UTC of the processed science image is also mentioned in the final output.

4 PIPELINE VALIDATION AND RESULTS

The pipeline was validated on ILMT images to ensure a seamless real-time execution when integrated with the incoming data. The execution time of the pipeline depends on factors like the image quality and crowdedness of the field. The median execution time per ILMT frame (dimensions 4K|$\times$|36K) is approximately 29 minutes when computed using the Intel^® Xeon^® Silver 4210R 10-core CPU clocking at 2.40 GHz. With an estimated acquisition of 35 frames per night, the total processing time for a complete night’s data set is nearly 17 h. The performance of the three main steps of the pipeline, namely image subtraction, transient detection, and transient candidate classification, were evaluated using individual tests.

4.1 ILMTDiff

The two aspects that the ILMTDiff image subtraction module was examined for were execution speed and quality of subtraction. The execution time for the ILMTDiff was optimized by fine-tuning the parameters to adapt to the ILMT images. The average processing time for 4096 |$\times$| 4096 pixel cut-outs of ILMT image was |$\sim$|16–20 s. For comparison, the High Order Transform of Psf ANd Template Subtraction code (hotpants) image subtraction software (Becker 2015) took similar time scales when tested on those images. It was also observed that the quality of subtracted images rendered by ILMTDiff was comparable to that by the hotpants software (Fig. 11) for most pairs of images. Comparable performances were obtained for Optimal Image Subtraction (ois; Sánchez et al. 2020) and properimage (Sánchez et al. 2019) which are two publically available python-based image subtraction codes. However, it should be noted that the performance of ILMTDiff depends on the choice of module parameters, which were determined after sufficient experimentation. In addition to the ILMTDiff, these other available codes can be integrated with the pipeline in future with an option to select among them.

4.2 TransiSearch

The evaluation of the TransiSearch module involved obtaining performance metrics for detecting transient/variable sources in the ILMT images. The adaptive detection technique described in Section 3.5 was implemented to mitigate the false positive detection. For a quantitative evaluation of the effectiveness of this technique, the pipeline was implemented on 30 i|$^{\prime }$|-band images acquired on 2022 October 28–30, both with and without implementing the adaptive detection technique.

The version of the pipeline without implementing the technique produced 488 detections, of which 318 were true positives and 170 were false positives, resulting in a precision of 0.65. In contrast, the adaptive detection method yielded 320 detections, with 290 true positives and 30 false positives, yielding a precision of 0.91. It should be emphasized that the precision in this context differs from one calculated using an artificially curated and balanced test data set. In transient survey images like the ILMT, the ratio of the real to bogus samples is highly imbalanced as the number of bogus samples outnumbers the real transients by a significant margin.

The two CNN models employed for the adaptive detection technique were separately evaluated. The validation accuracy of high-precision and high-recall CNN classifiers were 98.96 per cent and 97.71 per cent, respectively. The receiver operating characteristic (ROC) curve and the precision-recall (PR) curve for the two CNN models are shown in Fig. 12. The area under the ROC curve (AUROC) score for the high-recall classifier was determined to be 0.998, while that for the high-precision classifier was 0.997. The area under the PR curve (AU-PR) score for the high-recall classifier was 0.997, and that for the high-precision classifier was also 0.997. Confusion matrices for the high-recall and high-precision classifiers are given in Tables 5 and 6, respectively. The test accuracies were determined to be 98.07 per cent and 93.97 per cent, respectively. The confusion matrices, the ROC, and the PR curves were generated using a separate test data set with artificially added noise. Fig. 13 lists some candidates with corresponding CNN scores.

The sensitivity of transient detection can be evaluated by determining recall values for ‘real’ test candidates with varying S/N ratios. The test candidates were simulated with Gaussian profiles, by varying FWHMs and varying levels of artificial noise (to control the S/N ratio). All the candidates were then passed through the ‘real/bogus’ classifier and threshold-based filtering. The mean recall for 200 candidates per S/N ratio was plotted and is shown in Fig. 14. As expected, the high-recall classifier plateaued towards high-recall values for low S/N ratio samples, while the high-precision classifier showed a relatively steady increase.

To evaluate the sensitivity of the overall pipeline on actual images, its detection completeness for visually inspected asteroids in six ILMT frames was determined. The V-band magnitudes of the detections were queried from the MPC catalogue of minor planets. A total of 293 MPC asteroids were confirmed visually. 159 of them were fainter than 21 mag. 231 of the 293 asteroids were successfully recovered by the pipeline. Fig. 15 illustrates the recall values for different V-band magnitude ranges. Furthermore, Fig. 16 presents the MPC obtained V-band magnitude distribution of the asteroids identified in real-time with the pipeline during its operation. From the distribution, the median V-band magnitude of the detected asteroids was determined to be 19.80, with the 5^th percentile magnitude at 17.80 and the 95th percentile magnitude at 21.0.

4.3 NovaNet

To evaluate the NovaNet module, classifications performed only on valid detections (as determined by visual inspection) by the TransiSearch module were considered. Table 7 illustrates the predicted class distribution for valid detections with corresponding true positive class distribution at various common LST fields over eight nights in 2022 October–November. The test data set was supplemented with a few artificially generated ‘SN-like’ sources to account for the corresponding sample scarcity in the original data set of detected sources. Such sources were generated by embedding point sources in a few galaxies in the ILMT frame. Table 8 shows the confusion matrix for the classifications performed on the final test data. The classification accuracy was evaluated from the confusion matrix using the given formula and was determined to be 0.986.

4.4 Discovery of the new transients AT 2023yjc and 2024fxn

The third observation cycle of the ILMT commenced in 2023 November. Using frames acquired in previous epochs as references, 12 ‘SN-like’ transient events were detected using the pipeline in 761 analysed full-frame images from this cycle. The transients AT 2023yjc (Pranshu et al. 2023) and 2024fxn (Pranshu et al. 2024b), detected on 2023 November 13 and 2024 April 5, respectively, were reported as discoveries to the TNS. The names of the other detected transients include AT 2023vhj, 2023xow, 2024ccg, 2024ekk, 2024eab, and 2024fpx, alongside four confirmed SNe: SN 2023vcg, 2023wuk, 2023ngy, and 2024cjb. Six of the 12 detected transients are shown in Fig. 17. Fig. 18 shows examples of other variables and asteroids detected with the pipeline.

5 CONCLUSIONS

In this paper, we have discussed the PyLMT transient detection pipeline for the images acquired using the ILMT in the SDSS g|$^{\prime }$|⁠, r|$^{\prime }$|⁠, and i|$^{\prime }$| spectral bands. The image subtraction is performed using an in-built module called ILMTdiff while transient detection and candidate classification are executed inside the TransiSearch and NovaNet modules, respectively. The pipeline is equipped with a catalogue cross-matching step to assist in the search for new transients by rejecting known asteroids and probable variable stars.

The quality of image subtraction achieved with the ILMTDiff module, based upon Bramich (2008) approach, is comparable with other existing software like the hotpants. The precision value for real detections with the TransiSearch module, when implemented on ILMT frames, was determined as 0.91 and the test accuracy of the real/bogus classifiers ranged from 94 per cent to 98 per cent. Classification accuracy achieved with the NovaNet transient candidate classifier was 98.6 per cent. The pipeline has been successfully deployed on ILMT workstations and integrated with the data stream, leading to the detection of multiple transients, variable objects, and solar system bodies. The median execution time of the pipeline was determined to be approximately 29 min per ILMT frame. AT 2023yjc and 2024fxn were previously unreported transients discovered using the pipeline and subsequently reported to the TNS.

The ILMT science team regularly vets the candidates and interesting candidates are identified for follow-up. This potentially enables early-time identification of transients like SNe and their spectroscopic follow-up with other facilities. Such studies can be of great importance in enhancing our understanding of pre-explosion conditions of the SN progenitors, constraining the explosion mechanisms, and testing stellar evolution theories of massive stars.

ACKNOWLEDGEMENTS

The authors thank the referee, Kendall Ackley, for providing constructive comments on the manuscript, which improved the clarity of the presentation. The 4-m ILMT project results from a collaboration between the Institute of Astrophysics and Geophysics (the University of Liège, Belgium), the Universities of British Columbia, Laval, Montreal, Toronto, Victoria and York University, and Aryabhatta Research Institute of observational sciencES (ARIES, India). The authors thank Ankit Bisht, Hitesh Kumar, Himanshu Rawat, Khushal Singh, Nikhil Dharkiya and other observing staff for their assistance at the ILMT. The team acknowledges the contributions of AMOS (Advanced Mechanical and Optical Systems), CSL (Centre Spatial de Liège), Socabelec (Jemeppe-sur-Sambre) and ARIES’s past and present scientific, engineering and administrative members in the realization of the ILMT project. JS wishes to thank Service Public Wallonie, FRS-FNRS (Belgium) and the University of Liège, Belgium for funding the construction of the ILMT. ARIES thanks the Department of Science and Technology (DST), Govt. of India, for the realization of the project. PH acknowledges financial support from the Natural Sciences and Engineering Research Council of Canada, RGPIN-2019-04369. PH and JS thank ARIES for their hospitality during their visits to Devasthal. JS and KM acknowledge the assistance received from the Anusandhan National Research Foundation (ANRF, SERB-762 Visiting Advanced Joint Research (VAJRA) Faculty Scheme, India). KM, BK, BA, and ND acknowledge the support from the BRICS grant DST/ICD/BRICS/Call-5/CoNMuTraMO/2023 (G) funded by the DST, India. BA acknowledges the Council of Scientific & Industrial Research (CSIR) fellowship award (09/948(0005)/2020-EMR-I) for this work. MD acknowledges the Innovation in Science Pursuit for Inspired Research (INSPIRE) fellowship award (DST/INSPIRE Fellowship/2020/IF200251) for this work. KP and KM would also like to thank Ashish Mahabal and Stefano Valenti for their valuable advice and insightful suggestions.

DATA AVAILABILITY

To perform this work, the authors extensively used the images acquired with the ILMT during its commissioning period. The raw and processed images with astrometric calibrations are routinely made available to the public domain (Misra K. et al. 2024) and can be accessed using the URL https://cloud.aries.res.in/index.php/s/xPER9Y3XuaCsTL9. The publicly available survey images from the ZTF (Masci et al. 2019) were used to perform initial testing of the pipeline scheme, including key algorithms like image subtraction. The images can be accessed through the IRSA platform using the URL https://irsa.ipac.caltech.edu/applications/ztf/?__action = layout.showDropDown&. Other relevant data can be made available upon request to the authors.

REFERENCES

APPENDIX A: THE ilmtdiff IMAGE SUBTRACTION ALGORITHM

The process of image subtraction involves a few basic steps namely image alignment, background subtraction, flux matching, and PSF matching. Each of these steps as implemented in the ILMTDiff module is discussed below in detail:

• Science and Reference images: The reference image has to be subtracted from the science image to obtain the difference image. Transient detection is performed on this difference image. The reference image required for image subtraction should be acquired on a night with relatively better seeing conditions. Co-added reference images are desired to produce subtracted images with good S/N ratio. Limited imaging data was available during the pipeline validation stage. Hence, the current work used non-coadded single reference images acquired in good seeing conditions. With the acquisition of more data in future, a data base of co-added reference images will be constructed.

• Image alignment: The science and reference images may exhibit relative offsets due to imprecision in the timing of image acquisition. Additionally, due to the fixed-pointing nature of the telescope, the images are subject to Earth’s precession and nutation effects, causing systematic shifts along RA and Dec. The relative offset becomes more pronounced with increasing duration between the acquisition of science and reference images. Thus, the images are first aligned with the shift module from the scipy library, based on the WCS information in the FITS headers. Inaccuracies in astrometric calibration further influence the alignment of the two images. To address this potential inaccuracy, the module has a provision to align images using astroalign software (Beroiz et al. 2020).

• Background subtraction: The frames acquired with the ILMT exhibit uniform or spatially varying background illumination. This background is affected by factors such as moon phase, scattering of moonlight due to clouds, and even possible light leakage in the dome (now under control). These factors can cause a significant variation in background illumination across different ILMT frames. Therefore, the space-varying background is removed from the acquired images using the MedianBackground estimator available with photutils (Bradley et al. 2016).

• Flux matching: Depending on factors like cloud cover, presence of haze, reflectivity, and transmissivity of the telescope components etc., the total instrumental flux integrated for the sources in the image might differ from one night to another. Image subtraction requires integrated source flux in science and reference images (acquired on different nights) to be matched. The kernel optimization step in the ILMTDiff module itself scales the kernel with the appropriate flux scaling factor as a consequence of PSF matching. Nevertheless, provision has been made to calculate the scaling factor separately by evaluating the median of the aperture flux ratios of common sources in the science and reference images.

• PSF matching: PSF matching is the principle functional step of the image subtraction module. The images are acquired on different nights with possibly different seeing conditions. For successful image subtraction, it is crucial to match PSF between the sources in science and reference images. This is achieved by convolving the image with better seeing (smaller FWHM; ideally the reference image) with an appropriate convolution kernel. This kernel can be determined using least-squares optimization, which in this context, involves minimizing the sum of squares of pixel values for subtraction residuals in the difference images. The delta-basis formulation (refer to Section A1) was used as the principle kernel model for subtraction.

A1 Delta-basis kernel optimization for PSF matching

The algorithm begins by selecting a set of common sources in the science and reference images. These sources are above a specified detection threshold and are uniformly distributed across the image. The sources are extracted in square cut-outs of specified size, with a default value of 21 |$\times$| 21 pixels. This value can be adjusted with the stamp_size parameter. A 2D Gaussian profile is fitted to these cut-outs, allowing the determination of morphological parameters. A series of thresholds (refer to Table A1) are then applied to these parameters to filter ‘good’ sources for subsequent steps. The maximum number of sources used for kernel determination can be specified using the n_stamps parameter in ILMTDiff. A larger value for n_stamps parameter causes the algorithm to consume more time. Values ranging from 5 to 30 have been used for this parameter, yielding acceptable subtraction quality.

The source cut-outs in the reference image are convolved with a kernel with randomly initialized pixel values and are subtracted from the cut-out of the corresponding source in the science image. The kernel size can be specified by the user (default value is set at 7 |$\times$| 7 pixels). The pixel values of the kernel are iteratively modified using a numerical optimization method (Moré 1978; Virtanen et al. 2020) to get the optimal kernel. The sum of residuals of the subtraction between science image cut-outs and convolved reference image cut-outs is the objective function to be minimized. Mathematically, the residuals can be expressed using equations (A1) and (A2):

where |$S_{ij}$| and |$R_{ij}$| are the pixel values for individual science and reference source cut-outs, |$K_{kl}$| represents the pixel values for the convolution kernel (in kernel coordinates) and |$\sigma _{ij}$| is the total of background noise and Poisson noise contribution from science and convolved reference images. stamp_residual is the subtraction residual for an individual pair of science and reference source cut-outs (or stamps), while total_residual is the sum of all stamp_residuals, which has to be minimized. Kernel optimization automatically adjusts the kernel to account for the flux factor. In an optimized kernel, the sum of pixel values equals the flux scaling factor.

Upon determination of the optimal kernel, the reference image is convolved with this and is subtracted from the science image to get the subtracted/difference image. Fig. A1 illustrates the flowchart for the ILMTDiff image subtraction module.

APPENDIX B: TRAINING OF THE REAL/BOGUS CLASSIFIER

B1 Data set preparation

It is discussed in Section 3.5 that high-precision and high-recall CNN classifiers are alternatively used to reduce false positive detections effectively. The class distribution of the data set was kept different to construct the two different classifiers. The number of ‘real’ and ‘bogus’ samples was 2705 and 2527, respectively, for the high-recall classifier. However, the same was 1000 and 2527, respectively, for the high-precision classifier. A relatively greater number of ‘bogus’ samples in the training data set biased the high-precision classifier to be more sensitive in identifying bogus sources accurately and therefore have low overall false positive detections.

The ILMT data from the first commissioning cycle in 2022 October–November was used to construct the data set. The samples are square cut-outs of size 31 |$\times$| 31 pixels, corresponding to nearly 10 |$\times$| 10 arcsec² of the sky. The astrometric seeing of the ILMT can range from 1.5 to 3.5 arcsec. Also, there are occasional appearances of comatic flaring and astrometric misalignment of images (up to 1 arcsec). The cut-out size is sufficiently large to encapsulate the size and nature of the sources. Simultaneously, it is sufficiently small to constrain individual sources in the difference source cut-outs. The real sources were obtained by extracting cut-outs of ‘PSF-like’ point sources from acquired ILMT frames. Bogus samples were extracted from subtracted ILMT frames. Data augmentation by orthogonal rotations was utilized to enhance the model’s generalizability and the extensiveness of the data set. Special care was taken to ensure diversity in the visual appearances of the data set samples, enabling the training of robust networks.

B2 Data pre-processing and training

The samples in the data set had a wide distribution in scales of pixel values depending on source brightness. The models trained with such a data set can be unstable and perform poorly on real data. Therefore, it was decided to standardize data set samples using equation (B1):

where

Artificial noise was added to some of the data set samples to enhance the detection sensitivity of the trained model. The data set was split into training and validation sets in an 80:20 ratio.

The same CNN architecture was used for the high-precision and high-recall classifiers. The architecture consists of a feature extraction region with three convolutional layers, each coupled with a 2 |$\times$| 2 maxpooling. The model uses a combination of 2 |$\times$| 2 and 3 |$\times$| 3 pixel size filters. The fully connected region has 2 hidden layers. The activation function used in each layer is ReLU (Nair & Hinton 2010) except for the output layer where sigmoid activation function is used. The complete CNN architecture is shown in Fig. C1.

For training, the loss function used was binary-crossentropy and the optimizer was adam (Kingma & Ba 2015). Regularization techniques were also used to prevent overfitting. Two techniques used were dropout regularization (Srivastava et al. 2014) and L2 regularization. Dropout rates of 0.5 in the feature extraction region and 0.7 in the fully connected region appeared to work well. The regularization hyperparameter for the L2 regularizer was set to 0.02.

Additional data augmentation was performed by applying rotation_range, horizontal_flip, height_shift_range and width_shift_range using KerasImageDataGenerator module (Chollet et al. 2015). Then CNN architecture was written and trained as a sequential model using Google’s TensorFlow (Abadi et al. 2016) library. GPU mode available with Google Colab^® was used for training. After multiple instances of training, it was observed that the validation accuracy plateaued around 98 per cent in about 20 epochs with a default batch size of 32. The accuracy and loss curves for the high-recall classifier are shown in Fig. B1 and the same for the high-precision classifier are shown in Fig. B2.

APPENDIX C: TRAINING OF THE TRANSIENT CANDIDATE CLASSIFIER

C1 Data set preparation

The data set for the NovaNet transient candidate classifier was curated using the ILMT science frames acquired in the October–November cycle of 2022. The data set was created by extracting cut-outs from ILMT frames representing three different host situations (refer to Section 3.3): (i) ‘extended-host’, (ii) ‘point-host’, and (iii) ‘hostless’.

The ‘extended-host’ situation is characterized by a galaxy present in the reference image at the position of detection. ‘Point-host’ is characterized by a point-source and ‘hostless’ by the absence of any source. So to prepare the data set, appropriate cut-outs were cropped out from the ILMT images. For the ‘extended-host’ samples, it was ensured that the displacement of the galaxies relative to the cut-out centres mimicked the spatial occurrences of transients relative to its host galaxy.

A total of 407 samples for each category were cropped out. The samples were augmented with 90|$^{\circ }$|⁠, 180|$^{\circ }$|⁠, and 270|$^{\circ }$| rotations. This enhanced the total size and diversity of the data set. Finally, two separate data sets were created with sample cut-out sizes of 102 |$\times$| 102 pixels and 31 |$\times$| 31 pixels, respectively. They were used to train the CNN models of the two respective input sizes. Both data sets comprised 1628 samples for each class. Samples from the data set for ‘extended-host’, ‘point-host’, and ‘hostless’ classes are shown in Figs 7, 8, and 9 respectively.

C2 An ensemble of CNN models

The NovaNet module employs a logical combination of individual predictions from three CNN models to classify candidates. Depending on the cut-out sizes of images transferred to the CNN input, the models are one of two types: (1) one model with an input shape of 31 |$\times$| 31 pixels and (2) two models with an input shape of 102 |$\times$| 102 pixels.

The strategy of combining predictions from multiple classifiers proved effective in reducing false positive predictions for a specific class. For instance, to lower the false positive rate for ‘extended-host’ objects and expedite the search for SNe candidates, the prediction criterion was set to require a positive detection from all three classifiers. The flowchart in Fig. C2 illustrates this logical scheme for combining model predictions. The three classes are positioned at different levels in the flowchart depending on class priority (i.e. ‘extended-host’ objects are prioritized over ‘point-host’, followed by ‘hostless’).

The 31 |$\times$| 31 pixel classifier and 102 |$\times$| 102 pixel classifier architectures have two and three convolutional layers in the feature extraction region, respectively. Each convolutional layer is coupled with a max-pooling layer of size 2 |$\times$| 2 or 3 |$\times$| 3. A combination of 2 |$\times$| 2 and 3 |$\times$| 3 filters are used. The fully connected region in both types of classifiers has three dense layers and an output layer with three dimensions, representing the three possible classes. The activation function used for all the layers except the output layer is ReLU and that for the output layer is softmax. Batch normalization layers are also used with each convolutional layer. The CNN architectures of both classifier types are shown in Fig. C3.

C3 Training of the CNN

For training the models, the loss function used was categorical_crossentropy while the optimizer was adam. L2 regularization and dropout regularization were used to prevent overfitting in both types of classifiers. The regularization hyperparameter for the L2 regularizer was kept at 0.01. The dropout rate was 0.3 for the feature extraction/convolutional region and 0.5 for the fully connected region.

Like the real/bogus classifiers, the samples from the data set were standardized before training and the data set was split into training and validation sets in an 80:20 ratio. Artificial noise was again added to the training samples for enhancing model sensitivity. The CNN architecture was written using Google’s TensorFlow for python as a sequential network. Further data augmentation was performed by applying rotation_range, height_shift_range and width_shift_range using KerasImageDataGenerator module. The training was done using NVIDIA^® T4 GPU available with Google Colab^®. The 31 |$\times$| 31 pixels model was trained for 200 epochs while the two 102 |$\times$| 102 pixel models were trained for 50 epochs each. The final validation accuracy achieved was approximately 96 per cent for the 31 |$\times$| 31 pixel classifier and around 93 per cent and 95 per cent for the two 102 |$\times$|102 pixel classifiers. The accuracy and loss curves for the 31 |$\times$| 31 pixel CNN classifier are shown in Fig. C4 and the same for one of the 102 |$\times$| 102 pixel CNN classifiers are shown in Fig. C5.