SULI Internship Fall 2024

Data-Driven Image Restoration Methods In Cryogenic Electron

Tomography

H. Jones,1K. Pande,1and J. Donatelli1

Lawrence Berkeley National Laboratory

(*Electronic mail: n.henryjones@gmail.com)

(Dated: 6 December 2024)

I. ABSTRACT

In recent years, technical and algorithmic advances have placed cryogenic electron tomography (CryoET) at the

forefront of structural biology with the ability to capture the cellular environment in situ and determine the structures

of macromolecular complexes at resolutions near the atomic scale. Despite these improvements, signal-to-noise ratios

in CryoET remain low, which limits the determination of novel structures and conformations, as well as the study

of pleomorphic systems. We undertake a rigorous assessment of modern data-driven algorithms for denoising and

constraining missing-wedge data to benchmark performance and investigate the potential introduction of structural ar-

tifacts. We show that the modern algorithms CryoCARE, DeepDeWedge, IsoNet, and Topaz-Denoise all improve SNR

and contrast while greatly varying in their denoising performance and missing wedge compensation on our simulated

data.

II. INTRODUCTION

Cryogenic electron tomography (Cryo-ET) is an electron

microscopy technique that reconstructs a three-dimensional

representation of the sample (a tomogram) at nanometer res-

olution by collecting a series of two-dimensional electron mi-

croscope projections of the sample (micrographs) tilted at reg-

ular intervals (a tilt series). A large variety of mathematical

and statistical tools for reconstruction have been developed

over the two decades as the imaging potential of Cryo-ET has

been realized.

Cryo-ET is the only technique able to capture three-

dimensional, sub-nanometer resolution images of cell regions

and even entire cells without requiring classical labeling or

staining methods. Electron microscopes require the sample to

be held in a high vacuum to minimize scattering from non-

sample species in the imaging chamber. To capture the re-

alistic in situ cellular dynamics under such a vacuum for the

duration of time required to capture a tilt series (between 20 -

30 minutes), the sample is frozen in non-crystalline, vitreous

ice1. Recently, technological improvements in microscope

stability and imaging hardware have continued to improve tilt-

series resolution, pushing researchers to further reﬁne the re-

construction and downstream analysis tasks.

The primary difﬁculties in Cryo-ET reconstruction are the

well known "missing-wedge problem", accurate and efﬁcient

correction for the contrast transfer function (CTF), and the low

signal-to-noise ratio (SNR) inherent to electron microscopy.

Due to the high effective thickness of the sample at extreme

tilt-angles, tilt-series are generally collected up to ±60◦from

the in-plane tilt axis. This lack of information for higher tilt

angles is known as the missing wedge problem (it is visual-

ized as a missing wedge of spatial frequency information in

Fourier space) and creates artifacts in the reconstructed vol-

ume. As electrons move through the sample and are scattered,

the constructive and destructive interference resulting from

differences in quantum-mechanical phase leads to anisotropic

sampling of the spatial frequency components. This corrup-

tion is modeled by the CTF which can then be used to cor-

rect images. In practice, due to tilting of the sample during

projection acquisition and the difference in particles of inter-

est’s locations along the optical axis, ideal 3D CTF correc-

tion is challenging. As a result, CTF correction is often only

done in 2 dimensions and 3D corrections are often inefﬁcient2.

Lastly, due to technological and algorithmic limitations, to-

mograms suffer from non-stationary (Poisson-like) detector

noise and the structured noise resulting from the interpola-

tion and non-uniform undersampling of the three dimensional

Fourier space.

The last decade has seen a variety of technical and method-

ological techniques bringing scientists closer to delivering

cryoET’s promise of atomic scale renderings of in situ molec-

ular machinery.3In particular, subtomogram averaging (STA)

of identical particles and complexes of interest in a sample has

allowed for novel, atomic-level structure determination. How-

ever, with the low SNRs of reconstructed tomograms, STA is

most effective with a puriﬁed sample or a sufﬁcient number

of identical particles in situ. These constraints signiﬁcantly

limit the study of pleomorphic cellular objects, the cellular

ultrastructure, and the discovery of novel complexes, confor-

mations, and cellular mechanisms. As such, denoising as a

post-processing step is critical for the next stage in cryoET’s

development.

Modern denoising techniques for electron micrographs and

tomograms remain largely cosmetic as resolution improve-

ments don’t provide mechanistic insight3. Historically, image

denoising has been accomplished through linear ﬁltering tech-

niques such as simple low-pass, Wiener ﬁlters, and Gaussian

ﬁlters. Due to the equivalence of convolution in real space

and multiplication in Fourier space granted by the convolu-

tion theorem, linear ﬁlters are often applied in the frequency

space where computation is cheaper. While they beneﬁt from

simplicity, linear ﬁlters often only operate only locally and/or

require statical assumptions on the noise which are too sim-

SULI Internship Fall 2024 2

plistic for the composition of noise present in CryoET.

Non-linear methods such as median ﬁltering, nonlinear

anisotropic diffusion (NAD), non-local means (NLM), as well

as intensity and range ﬁlters require application (convolution)

in the image domain but can account for more sophisticated

noise and non-local patterns in an image. Denoising meth-

ods based on the wavelet transform also exist and have shown

promising results recently4. Despite these improvement, such

methods again require design based on prior assumptions and

are often designed with for applications with simpler noise

models than CryoET, where the composition of noises and

structured artifacts from the missing wedge problem signiﬁ-

cantly limits the success of modern ﬁltering techniques.

Since the advent of machine learning, scientists have lever-

aged the assumption-free paradigm of denoising by super-

vised neural networks, where the large parameter space of net-

work weights is traversed to predict clean target images from

noisy input images. While effective for some computer vi-

sion tasks, supervised denoising is ineffective for CryoET due

to the lack of clean target images and heterogeneous nature

of the sample. Thus, self-supervised machine-learning algo-

rithms or other data-driven statistical simulation methods have

the potential to vastly outperform existing denoising methods

for CryoET.

With the introduction of the Noise2Noise machine learn-

ing framework in 20185, it has now been shown that a typ-

ical neural network training task can be removed of its de-

pendency on clean target data, where the network instead

learns to map from one data-independent noisy realization

to another. The Noise2Noise paradigm has inspired new

data-driven methods for both reconstruction and denoising.

CryoCARE6, DeepDeWedge7, IsoNet8, and Topaz-Denoise9

demonstrate the beneﬁts of Noise2Noise-based models for

Cryo-ET where the acquisition of high-quality target images

required for earlier machine learning-based image restoration

tasks is no longer required.

A comprehensive review and comparison of these methods

at both tomogram and particle scales is missing in the litera-

ture which is the purpose of this paper(as well as subsequent

work). In Section III, we give an overview of the four pri-

mary models tested. In Section IV we give qualitative and

quantitative results for our best-case experiments. We show

improvement in SNR and contrast as well as intra-model FSC

and 3D FSC calculations to assess model consistency and di-

rectional anisotropy. In Section V we discuss our ﬁndings and

further directions.

III. METHODS

Here we provide an overview of the Noise2Noise frame-

work and the four tested Noise2Noise-based denosing and

missing wedge compensation methods.

A. Noise2Noise Frameowork

The Noise2Noise framework is based off the observation

that the typical training task for noisy-clean input-target pairs

(x′,y)under the L2 loss with network function fθ(x′)given by

argminθE(x′,y)|| fθ(x′)−y||2

2,(1)

minimizes to the following:

argminθ∑

x′

∑

L(fθ(x′),y)pX′,Y(x′,y)

=argminθ∑

x′

∑

L(fθ(x′),y)pX′,Y(x′,y)

=argminθ∑

x′

∑

L(fθ(x′),y)pX′(x′)pY|X′(y|x′)

=argminθ∑

x′

pX′(x′)∑

L(fθ(x′),y)pY|X′(y|x′)

=argminθEx′{Ey|x′{L(fθ(x′),y)}}.

Thus the optimal network parameters θremain unchanged

across all training samples if the input-conditioned target dis-

tributions pY|X′(y|x′)are replaced with arbitrary distributions

with the same conditional expectation. That is, for a dif-

ferent noisy realization x′′ of the same underlying signal y,

Ey|x′{L(fθ(x′),y)}=Ey|x′′ {L(fθ(x′′),y)}, which allows Equa-

tion 1 to be written as

argminθE(x′,x′′)|| fθ(x′)−x′′||2

2,

where the dependency on clean target data yhas been re-

moved. In the case of CryoET, splitting the tilt-series by ac-

quisition number into even and odd, data independent tilt se-

ries provides two distinct, noisy realizations of the same un-

derlying signal.

We note that while denoising can be applied before or af-

ter reconstruction, post-reconstruction denoising is preferred

for several reasons: (i) it does not affect the linear relation-

ship between projections and the reconstruction granted by

the Fourier slice theorem, (ii) it has a non-preferential view-

ing orientation unlike micrograph denoising, and (iii) it min-

imizes the risk of compounding denoising artifacts through

reconstruction.3

1. CryoCARE

In our testing of CryoCARE we used the preferred T2T-

eoa protocol which splits the tilt-series into even and odd se-

ries based on acquisition number. The two independent tilt

series are used to reconstruct two independent noisy tomo-

grams, from which subtomograms corresponding to the same

coordinates are extracted as input-target pairs for model train-

ing. Then a U-net model learns the distribution of the noise

across all subtomograms during training. To produce a ﬁnal

denoised volume, the trained model predicts the average of

the even and odd tomogram and the two denoised predictions

SULI Internship Fall 2024 3

FIG. 1. CryoCARE Processing

are averaged to produced the ﬁnal denoised volume (Fig. 1).

Note that cryocare operates solely in the image space and does

not consider the missing wedge problem.

2. IsoNET

IsoNET aims to reconstruct the missing wedge of spa-

tial frequency information by applying an additional missing

wedge in frequency space to the input subtomogram after in-

put and target subtomograms have been rotated.

After subtomogram extraction, subtomogram pairs are ro-

tated in the hopes that the model learns a rotation-invariant

understanding of the missing wedge. Presumably in an ef-

fort to limit the effects of interpolation, the authors chose to

apply the 20 unique kπ/2 subtomogram rotations that do not

repeat the z-x axes missing wedge in the original tomogram.

After rotation, an artiﬁcial missing wedge with the same ori-

entation as in the un-rotated subtomogram is applied to the

target image for all pairs. While this artiﬁcial missing wedge

encourages the model to "generate" or "reconstruct" the miss-

ing information, the target image is further compromised. To

retain and improve the original structural information, the au-

thors implement an iterative approach where after each train-

ing iteration, the U-Net model predicts each subtomogram and

the predicted missing-wedge information is substituted in the

frequency domain in place of the originally empty missing

wedge. These hybrid missing-wedge compensated subtomo-

grams are then used in the next found of training. In this

way the model iteratively learns to reconstruct the missing

wedge using input-target data increasingly close to ground

truth missing-wedge-free subtomograms.

The denoising module of IsoNet is optional and is based

on the Noisier2Noise and NoisyAsClean frameworks8. If en-

abled, in each IsoNet Reﬁne iteration (Fig. 2), a pure noise

3D volume is reconstructed by FBP of a series of 2D images

containing only Gaussian noise for each subtomogram. These

(a)

(b)

FIG. 2. (a) The IsoNet workﬂow, (b) the Reﬁne step in Isonet where

Noise2Noise training denoises and ﬁlls the missing wedge.

noisy volumes are then added only to the U-net input subto-

mogram so the network simultaneously learns to denoise and

reconstruct the missing wedge.

3. DeepDeWedge

DeepDeWedge aims to provide the missing wedge recon-

struction of IsoNet and the denoising of cryoCARE in a sin-

gle end-to-end deep learning model. The tilt series is split

into even and odd series based on acquisition number, each of

which is used to reconstruct and independent tomogram. After

extracting the corresponding subtomogram input-target pairs,

DeepDeWedge aims to learn the missing wedge by randomly

rotating sub-tomogram pairs and applying an additional miss-

ing wedge mask to the input subtomogram in the frequency

domain before returning to the image domain for training. In

this way the model is forced to learn to reconstruct the artiﬁ-

cially imposed missing wedge so that it may predict the single

missing wedge in the prediction task (Fig. 3).

4. Topaz Denoise

Topaz uses a similar U-net architecture to the previous mod-

els with a similar training scheme shown in Figure 4. In-

SULI Internship Fall 2024 4

FIG. 3. DeepDeWedge Processing. Note that the DeepDeWedge

authors shows 2D tilt series images as 1D Fourier slices and 3D to-

mograms as 2D objects in the Fourier domain.

FIG. 4. Training scheme for Topaz-Denoise. Note that for the au-

thor’s 3D extension of their initial model, the micrographs are re-

placed by reconstructed tomograms tomograms in the ﬁgure.

stead of training a model ourself, the authors provide a model

trained on 32 different tomograms for over a month that we

used for our study to examine the potential of "sample agnos-

tic" denoising models.

IV. RESULTS

A. Metrics

We present qualitative comparisons in resolution and con-

strast in image space as well as missing wedge compensation

in frequency space. To assess model consistency and the res-

olution of data-independent half-maps Gand Hwe use the

standard Fourier Shell Correlation (FSC) given by

FSC(r) = ℜ{∑ri∈rF[G](ri)·F[H](ri)}

q∑ri∈r|F[G](ri)|2+∑ri∈r|F[H](ri)|2,

where ris the frequency bin.

In order to investigate the directional anisotropy of de-

noising and missing wedge compensation, we computed 3D

FSC10 for model half maps. The 3D FSC computes a local

FSC for every point in frequency space, correlating over a

limited angular range (θ) of neighboring Fourier coefﬁcients

in the same Fourier shell. The output is a 3D analogue of

the FSC which contains rich information on the relative direc-

tional anisotropy of two tomograms, allowing for a more rig-

orous understanding of the effects of denoising and missing

wedge compensation on tomogram resolution and the poten-

tial introduction of structural artifacts. The 3D FSC at a point

kin Fourier space is given by:

FSCθ(k) = ℜ{∑|k′|=|k|,|k′·k|≥cosθF[G](k′)·F[H](k′)}

NG,θ(k)NH,θ(k),

NG,θ(k) = s∑

|k′|=|k|,|k′·k|≥cosθ

|F[G](k′)|2,

NH,θ(k) = s∑

|k′|=|k|,|k′·k|≥cosθ

|F[H](k′)|2.

We note that because we are using the pre-trained Topaz-

Denoise model, we are unable to generate half-maps in the

same way that we are for the other models which we train our-

selves using random halves of the extracted subtomograms.

Thus we omit Topaz from FSC comparisons.

B. Qualitative Denoising Results

Our results show that all tested denoising algorithms qual-

itatively improve SNR for certain cases and can signiﬁcantly

improve the ability to distinguish particles from noise. Fig-

ure 5 shows such best-case scenarios for the denoising tools

tested. At the 50 electrons per square angstrom (e/Å2) dosage

level, we observe the best qualitative results with CryoCARE

and DeepDeWedge, followed by Topaz. Topaz’s missing

wedge compensation and denoising does not qualitatively im-

prove SNR in the reconstrution, and the IsoNet on CryoCARE

case is visually comparable to the CryoCARE result. In-

terestingly, Topaz greatly outperforms the other tested mod-

els on the 12.5 dosage case, with CryoCARE showing the

next best results. DeepDeWedge and IsoNet appear to ex-

cessively smoothen the tomogram which make particles in-

distinguishable from noise and ice. Topaz appears to show

the best results on 2.5 dosage case, where all other images

are difﬁcult to interpret. We found that at our high dosage

case, the default noise arguments used by IsoNet’s models

worked well, however for the lower dosage cases we had to

reduce the default noise additions from {0.05,0.15,0.2,0.25}

to {0.05,0.075,0.1,0.125}.

Figure IV B shows a central slice of the shifted Fourier

transform of the tomograms shown in Figure 5 to illustrate the

differences in Fourier coefﬁcient intensity and missing wedge

compensation. We see that IsoNet appears to demonstrate

SULI Internship Fall 2024 5

FIG. 5. Qualitative results of denoising seen as a central zaxis tomogram slice for CryoCARE, DeepDeWedge, IsoNet, IsoNet on CryoCARE,

Topaz Denoise at three levels of dosage. The blue and red boxes highlight the locations of two ribosomes for comparison.

the most missing wedge compensation of any of the models,

though particularly for the 12.5 and 3 e

2dosage cases where

denoising perforamce was particularly poor. In the frequency

space, DeepDeWedge gives results closest to the noisy "Orig-

inal" tomogram, and appears to only effective compensate for

the missing wedge in the 50e

2dosage case. Despite demon-

strating superior denoising results for the lower dosage cases

in Figure 5, Topaz results in the frequency space appear simi-

lar across dosages and have more prominent streaks along the

zaxis.

Interestingly we found signiﬁcantly different denoising per-

formance for several models including CryoCARE depending

on whether the noisy tomogram was CTF corrected in IMOD

with the NOVA 3D CTF correction2. Surprisingly we found

that CryoCARE produced qualitatively improved results with-

out CTF correction 7.

C. Half-Map FSC Results

FSC results for the 50e

2dose case for data-independent half

maps show DeepDeWedge and IsoNet reconstructions agree

to resolutions exceeding the Nyquist frequency, while Cry-

oCARE and IsoNet on CryoCARE(Ionc) have resolutions ap-

proximately 0.15

pixel and 0 respectively (Fig. 8). CryoCARE

does no missing wedge compensation, so in comparison to

DeepDeWedge and IsoNet, its lower FSC resolution result is

expected. The IonC result is surprisingly low given the Cry-

oCARE result, as we would have expected the missing wedge

compensation of IsoNet to improve upon the FSC result of

CryoCARE.

Figure 9 shows the 0.5 and 0.143 contours of the 3D FSC

density map. With no missing wedge compensation, Cry-

oCARE shows the greatest degree of directional anisotropy

(perfect isotropy would be seen as circular slices through a

spherical 3D FSC extending to the Nyquist frequency).

Figure 10 shows all models preserve resolution along the

xaxis, with IsoNet and IonC showing the highest resolutions

along the yand zaxes respectively.

V. DISCUSSION AND FUTURE WORK

Our results show that modern denoising techniques all im-

prove the apparent signal to noise ratio and improve contrast

at varying degrees. The pretrained Topaz model appears to

perform best at the more experimentally realistic 12 and 3 e

dosage cases, despite doing very little missing wedge compen-

sation. Alternatively IsoNet appears to do signiﬁcant missing

wedge compensation, particularly at the lower dosages, but

is qualitatively the worst denoising method. This lack of a

clear relationship missing wedge compensation and qualita-

tive denoising results is very interesting and will be the study

of future work. The signiﬁcant effect of CTF correction on de-

noising performance is unexpected, as CTF correction should

reduce spatially correlated noise to enable better results. More

work is needed to study this effect.

Also of interest in the high FSC resolutions granted by

IsoNet despite having very poor denoising performance, per-

haps indicating superior model consistency despite poor per-

SULI Internship Fall 2024 6

FIG. 6. Qualitative results of denoising on the missing wedge seen through the central x-z slices of the logarithm of the magnitude of the

Fourier transform.

FIG. 7. Qualitative denoising results for CryoCARE with and with-

out CTF correction seen with central zslice of denoised tomogram

for the 12.5 dosage case after 200 epoch training.

formance. I ﬁnd the IonC result in Figure 8 counterintuitive as

well, as I expected IsoNet to reduce the directional anisotropy

of the CryoCARE denoised volume, granting the IonC result

a FSC curve at or above the CryoCARE FSC curve. With

IonC falling below the 0.5 and 0.143 FSC thresholds so fast, I

am led to believe that IsoNet was ill-conditioned for missing

wedge compensation on top of the high resolution CryoCARE

results. Perhaps this suggests further reducing the noise argu-

ments of IsoNet denoising to limit noise accumulation.

Results from 3D FSC calculations shown in Figure 9 are

somewhat difﬁcult to interpret, but again suggest a confusion

relationship between FSC results and denoising performance

as IsoNet shows minimal directional anisotropy, though Deep-

DeWedge appears the most consistent for the two thresholds

shown.

FIG. 8. Half-map FSC for CryoCARE, DeepDeWedge, IsoNet, and

IsoNet on CryoCARE (IonC).

FIG. 9. Central slices of 3D FSC results for CryoCARE, Deep-

DeWedge, IsoNet, and IsoNet on CryoCARE (IonC).

SULI Internship Fall 2024 7

FIG. 10. FSC along the x,y, and zaxes of the 3D FSC results whose

central slices are shown in Fig.10.

These whole-tomogram comparisons allow for an under-

standing of the qualitative SNR gains and consistency that

modern denoising techniques bring, however to determine the

structures of molecular complexes on an atomic scale, scien-

tists rely on subtomogram averaging to reﬁne molecular den-

sities. Therefore future work will leverage ground truth parti-

cle location and orientation data to achieve near-perfect subto-

mogram averaging with simulated data. Then, at the particle

scale, we can similarly use the FSC and 3D FSC to assess

these tools’ performance and their potential introduction of

structural artifacts.

After completing our analysis pipeline on simulated data,

we plan to conduct a similar analysis on experimental datasets

and use GAPSTOP template matching11 for experimental sub-

tomogram averaging.

ACKNOWLEDGMENTS

This work was supported in part by the U.S. Department

of Energy, Ofﬁce of Science, Ofﬁce of Workforce Develop-

ment for Teachers and Scientists (WDTS) under the Science

Undergraduate Laboratory Internship (SULI) program. We

conducted these model comparisons on simulated tomograms

using TEM-simulator12 in order to accurately benchmark re-

sults.

VI. REFERENCES

1L. Gan and G. J. Jensen, “Electron tomography of cells,” Quarterly Reviews

of Biophysics 45, 27–56 (2012).

2B. Turoˇ

nová, F. K. Schur, W. Wan, and J. A. Briggs, “Efﬁcient 3d-ctf cor-

rection for cryo-electron tomography using novactf improves subtomogram

averaging resolution to 3.4Å,” Journal of Structural Biology 199, 187–195

(2017).

3A. S. Frangakis, “It’s noisy out there! a review of denoising techniques

in cryo-electron tomography,” Journal of Structural Biology 213, 107804

(2021).

4X. Huang, S. Li, and S. Gao, “Applying a modiﬁed wavelet

shrinkage ﬁlter to improve cryo-electron microscopy imaging,” Jour-

nal of Computational Biology 25, 1050–1058 (2018), pMID: 29927624,

https://doi.org/10.1089/cmb.2018.0060.

5J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala,

and T. Aila, “Noise2noise: Learning image restoration without clean data,”

(2018), arXiv:1803.04189 [cs.CV].

6T.-O. Buchholz, M. Jordan, G. Pigino, and F. Jug, “Cryo-care: Content-

aware image restoration for cryo-transmission electron microscopy data,”

in ISBI 2019 (2019) pp. 502–506.

7S. Wiedemann and R. Heckel, “A deep learning method for simultaneous

denoising and missing wedge reconstruction in cryogenic electron tomog-

raphy,” (2024), arXiv:2311.05539 [cs.CV].

8Y.-T. Liu, H. Zhang, H. Wang, C.-L. Tao, G.-Q. Bi, and Z. H. Zhou,

“Isotropic reconstruction for electron tomography with deep learning,” Na-

ture Communications 13, 6482 (2022).

9T. Bepler, K. Kelley, A. J. Noble, and B. Berger, “Topaz-denoise: general

deep denoising models for cryoem and cryoet,” Nature Communications

11, 5208 (2020).

10Y. Z. Tan, P. R. Baldwin, J. H. Davis, J. R. Williamson, C. S. Potter,

B. Carragher, and D. Lyumkis, “Addressing preferred specimen orienta-

tion in single-particle cryo-em through tilting,” Nature Methods 14, 793–

796 (2017).

11S. Cruz-León, T. Majtner, P. C. Hoffmann, J. P. Kreysing, S. Kehl, M. W.

Tuijtel, S. L. Schaefer, K. Geißler, M. Beck, B. Turoˇ

nová, and G. Hum-

mer, “High-conﬁdence 3d template matching for cryo-electron tomogra-

phy,” Nature Communications 15, 3992 (2024).

12H. RULLGÅRD, L.-G. ÖFVERSTEDT, S. MASICH, B. DANE-

HOLT, and O. ÖKTEM, “Simulation of transmission electron micro-

scope images of biological specimens,” Journal of Microscopy 243,

234–256 (2011), https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-

2818.2011.03497.x.