3.1 Workflow to simulate single molecule FRET data

In order to obtain SMVs and FRET efficiency time traces that closely match experimental data, kinetic Monte Carlo (MC) simulations were used to generate N fluorescence intensity trajectories (Fig 2) [35]. For this purpose, we defined the total number of conformational states (molecular states) J as well as the corresponding J(J-1) monoexponential rate constants k_ij characterizing the interconversion between states i and j. Each state j was assigned to a discrete mean FRET efficiency FRET_j. Simulation of each time trace n and frame l in a time series characterized by a total length L, followed by generating SMVs, was achieved in eight steps:

1. Based on the kinetic rates k_ij, KMC simulations were performed to determine FRET(n,l) of each molecule n for each time bin l, i.e., the observation time a molecule spends in a certain molecular state. In the case of dwell times shorter than the time bin l, FRET(n,l) values of multiple states were averaged within the same bin l. Kinetic heterogeneity, i.e., state transitions characterized by multi-exponential kinetics [36], can be simulated by assigning different rate constants k_ij with the same FRET_j value but different states j. The kinetic model is further described in Section 3.2.
2. Donor and acceptor fluorescence intensities and were calculated according to Eq (2) based on the discrete FRET(n,l) values and assuming a total emitted donor fluorescence I_tot,0 in the absence of FRET. Even though a constant I_tot,0 for all SMs was used as default setting, it is possible to define molecule-to-molecule variations in I(n)_tot,0 to model non-uniform illumination or to create test data sets featuring multiple SNR values (Section 3.3.3). Further, the probability of donor excitation in absence of FRET was assumed to be independent of time and the relative dipole orientation of both fluorophores. Additionally, the exponential decay in the z-direction of the evanescent wave generated in TIR was neglected, as immobilization typically confines the biomolecules in close proximity to the surface. Differences between quantum yields QY_D and QY_A, as well as between the detection efficiencies η_D and η_A were accounted for by the correction factor γ (Section C in S1 File). It is noteworthy, that the gamma correction introduces an indirect change of the total emitted fluorescence (Table C in S1 File) independent of the user-specified distribution I(n)_tot,0 introduced above.
   A frequently observed phenomenon in smFRET are molecule-to-molecule variations. For example, variations can be observed with regard to the mean FRET value of a certain conformational state, the total emitted intensity and quantum yields [3,37]. We modeled cross-sample variability assuming a Gaussian distribution of the underlying VSP values characterized by a defined center and standard deviation σ, thus, FRET_j and σ_FRET,j, I_tot,0 and σ_Itot,0, and γ and σ_γ. This straightforward approach allows to simulate cross-sample variability in FRET states and trajectory SNR originating from instrumental artefacts, but also fluctuations of QY, interdye distances, local refractive indices, and fluorophore orientations that lead to κ² variations [18,38,39].
3. Instrumental imperfections resulting in direct acceptor excitation and spectral bleed-through were also considered. The respective correction factors are listed in Section C in S1 File [40,41].
4. To simulate spontaneous dye photobleaching (see option in Schematic A in S1 File), trajectories were truncated using an exponentially distributed photobleaching time.
5. N molecules were distributed over a virtual FOV in the respective donor and acceptor channel (Section 3.3.1) using either pre-defined or random coordinates (Section 3.3.2). The position of each molecule (x_n,y_n) is time-invariant, i.e., focal drift was assumed to be absent.
6. Pixel values of the respective single molecule coordinates were set to the calculated donor and acceptor fluorescence intensities I(x_n,y_n,l)_D/A in each frame and subsequently convoluted with the PSF of the virtual imaging system as discussed in Section 3.3.3. A spatially and/or temporally variable background was added to each frame to account for different sources of background as discussed in Section 3.3.4 and Section 3.3.5. Instrumental imperfections such as focal drift and chromatic aberrations can be simulated too.
7. Photon emission from fluorescent single molecules and background sources was implemented as all-or-none process with a constant probability to emit a photon according to Poisson statistics. In our MC simulations, the resulting photon shot-noise in donor and acceptor emission intensities was accounted for with a Poisson distribution of pixel intensities centered around the expected mean fluorescence intensity (Section 3.3.6).
8. The detected photon signal was finally convoluted with a realistic SNR function taking into account the sensitivity, linearity, and temporal noise of camera sensors according to "Emva 1288 Standard Release 3.1" [42] and Hirsch et al. [43]. (Section 3.3.6). All simulation parameters were set to default values inspired by values typically observed in an experimental setting (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Default values for the simulations carried out with the MASH-FRET simulation tool.

https://doi.org/10.1371/journal.pone.0195277.t001

3.2 Describing single molecule FRET trajectories as Markov chains

smFRET experiments rely on the assumption that a molecular system capable of adopting J conformations yields J unambiguously discernable FRET efficiencies. Therefore, we assigned each conformation j to a discrete FRET efficiency value FRET_j in our simulations. Here, the thermodynamic equilibrium between these states depends on the kinetic rates k_ij that characterize interconversion between state i and j for all states J. This interconversion was modeled as a Markov chain (Fig 2) [46]. In a Markov chain, state transitions are treated as homogenous processes, and the respective J × J matrix of transition probabilities p_ij to transit from a state i to j is defined by the transition rates k_i≠j and the frame rate f according to [47]: (3) where p_{i = j} determines the probability to stay within the same state and p_i≠j to transit from state i to state j, respectively. For i ≠ j, rates were assumed to adopt values ≥ 0, whereas rates were set to 0 for i = j. In the context of this study and for each simulated time trace n, we defined the state i of time bin l = 0 as the state with the highest probability p_{i = j} according to the probability matrix in Eq (3). We assumed a total length L of the observation time and determined the conformational state at each time bin by means of a KMC simulation. Please note that Matlab's built-in function for HMM time series generation (hmmgenerate) parametrized using p_jj’ in Eq (3) to build a time-resolved kinetic model and to generate the corresponding FRET time trace does not allow for time averaged states of single bins or frames, respectively. The simulation of the respective states in the following time bins can be carried out in two different ways: (i) bin wise or (ii) dwell time wise. For (i) we generated a random value from a uniform probability distribution between 0 and 1 and compared it to the cumulative probability vector of the respective state i in time bin l = 0. Thus, we defined the new state in time bin l = 1. The simulation was repeated for each time trace n until time bin L was reached. For (ii) we set p_{i = j} = 0, generated a random value and compared it to the probability vector . Thus, we obtained the next state j and generated the dwell time in state I from a monoexponential probability distribution, a hallmark of classical chemical kinetics, characterized by the respective rate coefficient k_ij. All time bins within the generated dwell time were set to state i. In the case of dwell times exceeding the last time bin, the last dwell time was truncated to the total length L. Note that in particular transitions rates k_ij ≥ f lead to the detection of artificial states due to time averaging [29]. Therefore, in the case of dwell times shorter than the binning time, we built a time-weighted average of consecutive states in time bin l and l+1 yielding an averaged FRET value (artificial states) typical for kinetics faster than the exposure time of the camera. Both approaches objectively define the conformational state j of the (bio)macromolecule under study n for each single time bin l. The number of simulation steps in the second approach is drastically reduced, which allows faster simulations. Therefore, it is recommended for the creation of long time traces, traces with small transition rates and traces with large differences between transition rates in order to perform a statistically meaningful kinetic analysis later on.

In intensity-based smFRET experiments, kinetic rates, and thus the transition probabilities, are experimentally accessible from dwell-time analysis [3] or by maximizing the likelihood of a matrix of rate constants in Eq (3) according to Gopich and Szabo [48]. Thus, experimentally obtained rate constants can be used to simulate the molecular system under study for comparison, an approach which was presented recently [24].

3.3 Instrument specific configuration

In smFRET, the detected signal originates from fluorescent molecules that are only a few nanometers in size. This results in a diffraction-limited spot, whose intensity profile is determined by the characteristics of the optical imaging system, including the microscope objective and beam expander that is typically placed in the detection pathway for magnification (please see Table C in S1 File for further examples). In the following, we describe how we accounted for these instrument specific parameters with an appropriate PSF model, by adjusting the number of single molecules in the FOV, the background signal and the noise model of the camera. All of these VSPs can be independently defined in our MATLAB GUI shown in Schematic A in S1 File.

3.3.2 Distributing single molecules within the FOV.

We used two approaches to distribute N simulated molecules within one half of the FOV: (i) We positioned molecules at predefined pixel locations (Fig 3A), yielding regular patterns similar to the ones obtained when nanostructured materials like zero-mode waveguides are used for surface-immobilization [50,51]. Here, overlap between individual PSFs is absent. (ii) We randomly distributed molecules (Fig 3B), typically leading to overlapping PSFs as observed when surface immobilization is achieved via biotinylated PEG or BSA [36,52].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. PSF and FOV simulation.

(A) Example of simulated PSFs with w_det,0 = 1 pixel (top) and w_det,0 = 2 pixel (bottom) and their appearance depending on their subpixel localization. (B) Representative averaged SMV featuring randomly positioned SMs and an inhomogeneous illumination profile w_ex,x,0 = w_ex,y,0 = 256 pixel.

https://doi.org/10.1371/journal.pone.0195277.g003

To obtain a single molecule density ρ, which is defined as the number of SMs per area, and to avoid overlap of different spots, the number of simulated molecules N was adjusted as a function of the image dimensions, i.e., the FOV. To simulate a typical single molecule experiment, we used a low surface molecule density (0.035 μm^-2 ≤ ρ ≤ 0.063 μm^-2). On average, this corresponds to 82 ≤ N ≤ 146 molecules distributed over the simulated FOV of 256 x 128 pixel (Fig 3B). Please note that we used subpixel accuracy to distribute molecules within the FOV, resulting in different appearance of single molecules in SMVs depending on their localization relative to the grid of pixels (Fig 3A). Subpixel accuracy in simulated SMVs is particularly important for the evaluation of spot detection algorithms in super resolution microscopy techniques [16,53].

3.3.6 Camera noise model.

Camera noise is composed of photon or photoelectron shot noise, amplification noise, spurious charge and camera-specific read-out noise, which contribute in an additive fashion [42,57,58]. Herein, we modeled EMCCD camera noise according to Hirsch et al. [43], taking only noise components into account, which are experimentally relevant: (i) We considered shot noise from all light sources contributing to photons reaching the detector pixel I(x,y,l)_D/A. The probability of n_ph incident photons to be detected on the detector pixel (x,y) during a frame l is given by a Poisson distribution P(n_ph;μ_ph = I(x,y,l)_D/A characterized by the mean number of incident photons μ_ph. The simulation of sole photon noise is commonly used for the simulation of single photon detection applications [19,45,59–62]. (ii) The second noise contribution is the camera shot noise σ_pe common to all (EM)CCDs or sCMOS [42,43]. The probability of detecting n_ph photons, i.e., producing n_pe photoelectron, is the quantum efficiency η of the detector already included via the gamma correction, and follows a binomial distribution. The joint probability of the Poisson distribution and the binomial distribution is again a Poisson distribution P(n_pe;μ_pe = ηI(x,y,l)_D/A characterized by the mean number of accumulated photoelectrons. (iii) During the read-out process, electrons are shifted through the pixel of the detector area, the readout-register and the EM register by means of changing electrode voltages. This shift creates unwanted electrons, a so-called clock induced charge (CIC) as one component of spurious charge. The probability of observing CIC is very low, though, CIC nonetheless adds non-negligible intensity spikes to the detected signal. These may introduce artefacts in SMV analysis. (iv) Thermal noise, as another component of the spurious charge, is usually suppressed by cooling the detector to very low temperatures (-80°C). Therefore, we did not consider it further. According to [43], the number of input electrons n_ie in the EM register is a convolution of the two Poisson distributions of photoelectrons and spurious charge P(n_ie;μ_ie = ηI(x,y,l)_D/A + CIC) (v) Amplification noise of an EM register with gain g was modeled as a gamma distribution G(n_oe;μ_oe = gμ_ie) where the mean number of output electrons μ_oe = gμ_ie consist of the Poisson distributed sum of all input electrons arriving in the EM register. (vi) Readout-noise σ_d results from the conversion of the analog electron signal, i.e., the number of output electrons n_oe, into a discrete image value μ_ic = μ_oe/s. We described readout noise with a Gaussian distribution N(n_ic;μ_ic = μ_ic,dark + μ_oe/s, σ_d) characterized by a standard deviation σ_d and the amplifier sensitivity or analog-to-digital factor s. We added a constant bias offset μ_ic,dark which is usually added electronically to avoid negative image counts. (vii) Quantization noise σ_q of the analog-to-digital converter was neglected, because moderate amplifier sensitivities are typically used. Thus, we yield the Poisson-Gamma-Normal (PGN) noise model of the EMCCD camera described earlier [43] (8)

In difference to reference [43], we found that the CIC noise probability is well described by an exponential distribution Exp(CIC) = A_CIC ⋅ exp(−I/τ_CIC) that features the terms A_CIC and τ_CIC (Fig 4). Here, A_CIC depends on the EM gain and has usually very small values whereas τ_CIC varies with the hardware binning and the vertical clock speed to shift the electrons to the read-out register. Further, we described the number of output electrons n_oe of the EM register with a normal distribution N(μ_oe = I(x,y,l)_D/Aη_D/Ag, σ_oe) with mean image counts μ_oe and variance σ_oe² = μ_oe. We approximated the PGN model by a Normal-Exponential-Normal (NExpN) model, where the weighted sum of a Normal distribution and the CIC noise was convoluted with the read-out noise: (9)

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Camera noise simulation.

Representative experimental camera noise of an Andor iXon3 DU 897D camera, following EMVA Standard 1288 notation [42]. (A) Standard deviation of camera noise as determined from single pixel temporal intensity fluctuations over 100 frames of single Cy3-labeled RNA with EM gain = 300, t_bin = 100 ms and a readout rate of 10 MHz. Excitation intensities were varied to yield mean signal intensity rates between 0 and 6000 image counts per frame. Fitting with Eq (13) yields: K = 57.7 IC e^-1, μ_ic,dark = 113 IC e, σ_d = 0.067 e and σ_q = 0 IC (B) SNR characteristics of the dark count corrected intensity signal in comparison to an ideal image sensor. The camera units (image counts) were converted into photon counts according to Eq (12). (C) Histogram of the experimentally observed image counts with closed shutter (dark image) for the characterization of CIC noise. Pixel intensities were collected from L = 100 video frames (512×512) using the same settings as in (A). Fitting with the NexpN model in Eq (9) resulted in μ_oe = 1069 IC.s^-1, A_CIC = 0.02 and τ_CIC = 205 IC s^-1. PGN noise model parameter (50000 samples): μ_ph = 0 pc, CIC = 0.02 e, others as determined in (A). N noise model parameter (50000 samples): μ_ph = 0.02 pc (D) Histogram of experimentally observed image counts of a single time trace (1000 frames) of Cy3 labelled RNA. PGN noise model (50000 samples): μ_ph = 85 pc, CIC = 0.02, others as determined in (A). Parameters of the NexpN noise model in Eq (9) and the Normal distribution in Eq (10) are chosen in the same way as for the PGN noise model.

https://doi.org/10.1371/journal.pone.0195277.g004

For sufficiently large EM gains and photon count rates CIC noise becomes negligible. Hence, the NexpN model simplifies to a Normal distribution with mean image counts μ_ic and variance σ_ic, which we called Normal (N) noise model defined by the parameter set I(x,y,l)_D/A,η_D/A,K and μ_ic,dark: (10)

In order to estimate the model parameters from experimental values, we recorded the signal of single surface-immobilized, Cy3 labelled RNA molecules with varying fluorescence intensity as described elsewhere [3,36]. The mean signal intensities μ_ic and standard deviations σ_ic are shown in Fig 4A. Following the nomenclature introduced in reference [43] and the standards for the characterization of image sensors and cameras [42], we defined the temporal variance σ_ic² of the digital signal as the sum of the aforementioned noise sources: (11)

Here, σ_d combines the noise related to the sensor read-out, the amplifier circuits and the spurious charge that is considered to be independent from the incoming photon signal. K = g/s is the overall system gain, i.e., a linear factor converting the charge into the digital output signal μ_ic as follows: (12) where μ_d denotes the mean number of electrons and μ_{ic, dark} = Kμ_d the mean grey or offset-value of a camera accessed experimentally in the complete absence of light (Fig 4A and 4C). The combination of Eqs (11) to (13) yields the variance of the image count signal of the camera assuming μ_pe = σ²_pe = ημ_ph: (13)

The determination of K from regression analysis as depicted in Fig 4A is known as photon transfer method [57,58]. K was further used to calculate the mean number of incident photons μ_ph as a function of the measured or simulated intensity signal of the camera μ_ic and vice versa in Fig 4B. With Eqs (12) and (13) one can write the SNR ratio of the camera signal as follows: (14)

Fig 4B shows the SNR comparison of a real and an ideal sensor. An ideal sensor is characterized by a detection efficiency η = 1, negligible dark σ_d = 0 and discretization noise σ_q/K = 0. Here, the SNR of the investigated EMCCD as a function of photon counts is close to the square root function of an ideal sensor. We validated the introduced noise models Eqs (8) to (10) by comparing them to experimental noise distributions in absence and presence of light (Fig 4 and 4D). Except for the simple N noise model in Eq (10) in case of dark images, we found all noise models to be in good agreement with the experimentally derived signal intensity distribution.

Since recently, low-noise sCMOS cameras featuring megapixel resolution achieve similar or even higher frame rates as their low-resolution CCD counterparts [11]. This allows for a greater SM sensitivity and larger FOVs given the same magnification of the optical system. Even though low-noise sCMOS cameras are devoid of amplification noise, the SNR is comparable to EMCCDs. All relevant VSPs related to the camera noise model are implemented in the GUI shown in Schematic A in S1 File.

4.1 Evaluation measures and classification

To compare different algorithms used in the context of SMV data analysis, we first classified the true values known from the input VSPs as ground truth (GT) (Fig 5A, panel I). The output of the algorithm was then classified as true positive (TP, Fig 5A, panel II), i.e., correctly detected values, false positive (FP, Fig 5A, panel III), i.e., erroneously detected non-existing values, false negative (FN, Fig 5A, panel IV), i.e., undetected existing values which are incorrectly classified as non-value, or true negative (TN, omitted in Fig 5A as the herein tested algorithms do not classify TNs), i.e., values correctly classified as non-value. We then quantified the performance of different algorithms by computing the precision (also: positive predictive value) and the recall (also called sensitivity) [63,64]: (15)

In addition, the detection efficiency was quantified using the accuracy: (16)

Recall, precision and accuracy adopt values between 0 and 1 and reach 1 when TP ≠ 0 and FP = FN = 0. Since most smFRET-specific algorithms do not produce a TN class output, TN = 0 in most cases. In the context of method evaluation, the accuracy can be used to optimize the input MP in order to assess the method-specific limits of its particular output parameters, to quantify maximum accuracies and to test computation times. Since it is generalizable and easily applicable to other algorithms, we believe that it may contribute to the standardization, and hence, the reproducibility of smFRET data analysis.

4.2 SMV test sets with different VSPs for method evaluations

We generated a number of SMVs which served as GT for the comparison of different algorithms. We classified the simulated SMVs depending on which VSP was varied and which SMV property was changed, respectively (Table D in S1 File). In particular, we varied the following VSPs: (i) The single molecule fluorescence intensity I_tot,0, which affects the SNR, (ii) the PSF width w_det,0, which mimics different imaging quantities, (iii) the molecule-to-molecule distances IMD and (iv) the molecule surface density ρ, both of which affect SM spot overlap, (v) the spatially variable background that affects the signal-to-background ratio, (vi) the trace length L and (vii) the ratio of forward/backward transition rate constants k_ij / k_ji, both of which affect the kinetic model determination and (vii) the heterogeneity associated with state-specific mean FRET values and the total emitted fluorescence intensity, both of which may influence the inferred model. All simulated SMVs are freely available for download at https://github.com/RNA-FRETools/MASH-FRET for the individual assessment of smFRET-specific algorithms.

4.3 Application of simulated SMVs to optimize single molecule detection parameters

To achieve their optimal performance, SM localization algorithms require optimized input MPs (Fig 1). We optimized the input MPs of four spot detection algorithms, Houghpeaks (HP), in-series screening (ISS), Schmied (SCH) and TwoTone (2tone) (Table 2), and used simulated SMVs of the category (i-ii) suitable for testing the ability of an algorithm to detect SMs given a heterogeneous distribution of the total emitted intensities I_tot,0. All four algorithms apply an intensity cut-off I_thresh to the SM image prior to SM localization. This ensures that the algorithm can be used at different signal-to-background ratios. All tested detection algorithms process SMVs either sequentially (frame-by-frame) or upon averaging over all frames (frame/time-averaged SMV). The algorithms differ, however, with regard to the number of input parameter, the size parameter related to the PSF width (HP and ISS) or make use of a band pass kernel width (2tone). Different classification algorithms for TP, FP, and FN have been presented: (i) the Greedy approach which classifies TPs using the nearest neighbor criterion (type III in Fig 5A), and (ii) the Gale-Shapley approach, also known as stable marriage problem, which aims at maximizing TPs (type IV in Fig 5A) [63,65]. We used the Greedy approach, which is easy to implement and commonly applied. The maximum number of detected spots per frame was fixed to an upper bound larger than the GT to allow for the detection of false positives The resulting values for TP, FP, and FN were used to determine the algorithm-specific combination of input MPs yielding maximum precision, recall, and accuracy (Eqs 15 and 16).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Spot detection algorithms tested.

https://doi.org/10.1371/journal.pone.0195277.t002

Fig 5C illustrates the optimization workflow for the home-written function ISS. Here, the algorithms-specific spot detection parameters I_thresh and NhoodSize were varied (left panel). Decreasing I_thresh improves recall (top row) but deteriorates precision (middle row), resulting from a concomitant increases of true and false positives. The maximum values for evaluation measures recall, precision and accuracy cover a range of MP, I_thresh and NhoodSize, depending on the input parameters. These values are highlighted in Fig 5C using red bounding boxes for the underlying SMVs of category (i) at I_tot,0 = 40 pc and w_det,0 = 1 pixel. At 40 pc all evaluation measures reach the maximum achievable value of 1, whereas the MP range for the maximum accuracy is more limited than for recall and precision. From this range, maximum accuracy is achieved exemplarily for I_thresh = 13 pc and NhoodSize = 9 pixels (Fig 5C, solid grey square). I_thresh is the more critical value for high recall values within the examined parameter space in category (ii) simulations co-varying I_tot,0 and w_det,0. The NhoodSize parameter of ISS has no impact on the mean recall, precision and accuracy for w_det,0 < 3 pixel and I_tot,0 > 80pcpf. However, for conditions of decreasing SM signal (per pixel), i.e. towards w_det,0 = 3 pixel and decreasing I_tot,0 below 80 pcpf, maximal precision and accuracy can only be achieved by decreasing I_thresh and increasing NhoodSize (Fig 5C, middle and right). Please note that there is a strong drop of the maximum accuracy upon decreasing I_thresh (Fig 5C, left).

In order to optimize the input parameters of the other spot detection algorithms assessed, both I_tot,0 and w_det,0, as well as the horizontal intermolecular distance IMD and w_det,0 were co-varied. The results are shown in Fig B in S1 File. For a constant fluorescence intensity, the parameter I_thresh has a greater influence on accuracy than NhoodSize, in particular if IMD >> w_det,0.. However, optimum values for I_thresh and NhoodSize depend on the signal intensity I(x,y,l), the camera offset μ_{ic, dark}, and the camera noise σ_ic, and are thus specific to the present set of simulation parameters assessed. The parameter optimization for 2tone is special, because we found two parameter regimes yielding optimal results. The band pass kernel size BpSize of 1 and 3 requires I_thresh parameters of 5.5 and 1.5, respectively. Such an anti-correlated effect between two parameters yielding maximum accuracy was demonstrated for ISS at increasing w_det,0 and decreasing I_tot,0 values too. Without going into methodical details, a modest increase of I_thresh usually reduces FPs, which can be a consequence of image artifacts brighter than the local background caused by image processing steps used in a SM detection method. 2tone shows a clear preference of BpSize = 1 for the I_tot,0 dependence which is not the case for the IMD dependence.

In summary, reducing only the parameter I_thresh in the four investigated SM localization methods primarily results in more TPs and FPs, and thus, higher recall but lower precision. In general, a PSF theoretically leaks out of the ROI defined by the neighborhood size parameter NhoodSize, making e.g. ISS and HP prone for the detection of FPs at the boundary of the Nhoodsize-ROI. SM localization methods may contain image processing steps (usually implemented as black-box) that entail similar PSF smear effect (e.g. kernel size of 2tone). Thus, highest accuracy is often a compromise between I_thesh and other parameters like Nhoodsize. In smFRET, for low molecule densities where PSF overlap is statistically rare, one might consider prioritizing precision over recall, a strategy to avoid duplicates (FPs located close to GT).

After parameter optimization, SM localization methods are compared in terms of recall, precision and accuracy using their individual set of optimized MP and varying SM total emitted intensity I_tot,0 (Fig 6A). To enhance performance contrast, the individual result for recall, precision and accuracy were ranked between the four assessed methods with 1 to 4 points. In case of equal performance, ranking points are equally shared between the respective methods (Fig 6B). Both methods 2tone and ISS have a low false detection rate and thus a high precision. This is is beneficial for the detection of SM spots and a subsequent SM trace generation to avoid “empty” time traces which contain only background signal. ISS achieved best values for accuracy, however, comparable with 2tone. In detail, we find ISS to perform better for higher and 2tone for lower intensity values I_tot,0 (Fig 6B). Therefore, ISS and 2tone are both recommended for SM localization in SMVs akin to category (i). Note, that for other categories evaluation results may differ. To provide a deeper insight into the parameter optimization and method comparison presented in this section, we refer to https://github.com/RNA-FRETools/MASH-FRET.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Evaluation of single molecule spot detection after MP optimization.

SM detection was performed with four different algorithms, ISS, HP, Sch and 2tone, using simulated SMV of category (i) (Table D in S1 File) with and w_det,0 = 1 pixel varying molecules total emitted intensity I_tot,0 form 1 to 300 pc. (left) Recall, precision, and accuracy values of exemplarily chosen I_tot,0 values. (right) Ranking of the four algorithms for recall, precision, and accuracy.

https://doi.org/10.1371/journal.pone.0195277.g006