SULI Internship Spring 2024

Particle-in-Cell, Magnetic Reconnection Analysis and Exascale Computing

visualization tool development

H. Jones,1Dr. R. Jambunathan,1Dr. A. Myers,1Dr. H. Klion,1Dr. A. Nonaka,1Dr. A. Huebl,1and Dr. W.

Zhang1

Lawrence Berkeley National Laboratory

(*Electronic mail: n.henryjones@gmail.com)

(Dated: 25 April 2024)

I. ABSTRACT

Mulitiphysics simulations at the scale of modern high performance computing requires optimizing both simulation

run time and post-processing efﬁciency. This paper details efforts contributing to both of these problems: a study of

mesh-reﬁnement techniques for relativistic magnetic reconnection using the particle-in-cell code WarpX is presented

in parallel with the development of a yt-project front-end that uses the backend-ﬂexible openPMD-api for data I/O.

Relativistic magnetic reconnection is a process that drives non-thermal particle acceleration in many astrophysical

settings. Efforts to optimize simulations of magnetic reconnection with mesh-reﬁnement have had limited success

due to a variety of physical circumstances which present novel challenges for existing mesh-reﬁnement techniques. We

explore a variety of mesh-reﬁnement parameters and introduce novel features in WarpX to demonstrate the accuracy our

simulations on modern HPC systems. Our downstream analysis of simulation results with the yt-project motivates our

work bringing mesh-reﬁnement capabilities to the yt-project for simulation data with the openPMD meta-data markup.

II. INTRODUCTION

As modern supercomputers approach the Exascale (1018

64-bit operations per second) in computational power, the

complexity of modern multiphysics simulations demand

highly optimized codes and memory-efﬁcient routines for data

processing. The DOE-funded AMReX solver framework1is

particularly powerful on the world’s largest supercomputers

and provides the foundation for a variety of high performance

computing (HPC) multi-physics codes, including the highly

parallel, Particle-In-Cell (PIC) code WarpX used for our study

of relativistic magnetic reconnection2.

Magnetic reconnection is a fundamental kinetic process

that drives non-thermal particle acceleration leading to high-

energy radiation in many important astrophysical systems in-

cluding solar ﬂares, gamma-ray bursts, and pulsars. Dur-

ing reconnection, strong anti-parallel magnetic lines in close

proximity break and reconnect, rapidly converting magnetic

energy to particle kinetic energy.

To explore the kinetic mechanism of magnetic reconnec-

tion, a natural and well-established method of choice is

Particle-In-Cell method (PIC), wherein the simulation evolves

according to ﬁrst-principles to capture relativistic particle mo-

tion and its interaction with electromagnetic ﬁelds. At its core,

each time step of a PIC simulation involves four steps: 1) Par-

ticle evolution according to the Newton-Lorentz equations, 2)

Particle deposition on meshes by interpolation, 3) Maxwell’s

wave equations (for electromagnetic systems) are evolved on

the ﬁeld grids, 4) Field grids are interpolated back onto the

particles to repeat the cycle.

Mesh reﬁnement (MR), referring to the mesh resolution,

allows for focusing computational resources on particular re-

gions of interest in the simulation domain and has been proven

to provide impressive speedups while maintaining accuracy

in ﬂuid-ﬂow problems. Due to electromagnetic waves caus-

ing non-physical artifacts at the coarse mesh-ﬁne mesh inter-

face, MR is challenging for electromagnetic PIC, and most

studies of magnetic reconnection have been limited to two di-

mensions due to computational constraints. Building on the

work of Klion et al. 3, this paper documents our investigation

of the key parameters and mechanisms by which relativistic

magnetic reconnection can be well resolved with MR using

the WarpX code.

In parallel with this work, I contributed to the open-source

visualization and analysis library the yt-project4. The yt li-

brary allows for memory-efﬁcient handling of large simula-

tion datasets for a variety of ﬁle formats including HDF5 data

with the lightweight openPMD standard meta-data markup5.

The existing openPMD frontend of yt has been reworked to

use the openPMD-api for data I/O, allowing for the reading

of ADIOS2 ﬁles in addition to HDF5 ﬁles while bringing MR

visualization capabilities.6

III. SIMULATION SETUP AND DESIGN

A. Relativistic Magnetic Reconnection

1. Harris-Like Sheets

The two dimensional simulation setup is built upon the

work of Klion et al. 3which can be visualized by the

schematic in Fig. 1. Two anti-parallel, Harris-like current

sheets7are centered at ±xc=±Lx/2 where Lxis the half-

width of our domain along the xaxis. The sheets are orthogo-

nal to our periodic domain in the x−zplane. Two sheets are

required for the periodic domain due to opposite signs of the

background magnetic ﬁeld on either side of one sheet. Our

unit of time will be the inverse upstream electron gyrofre-

quency, ωc

−1≡me/(eB0), where the speed of light is c, the

SULI Internship Spring 2024 2

−Lx

−xc

−LzLz

2δ

ﬁeld gather buffer absorbing layer

FIG. 1. Initial conﬁguration for two-dimensional relativistic mag-

netic reconnection with mesh reﬁnement. The initial current sheets

(orange and purple) have half-widths of δand are located at x=±xc.

The difference in grid density demonstrates the location of the two

reﬁnement patches. Along the xaxis boundaries of each reﬁnement

patch, there is an absorbing layer (dark green hatched) and a larger

ﬁeld gather buffer (FGB) region (green highlight). The grid lines and

other features on this schematic are not shown to scale.

elementary charge is e, and the electron mass is me. Unless

otherwise speciﬁed, Table 1 of Klion et al. 3details the re-

maining choices of parameters such as mass magnetization,

the base unit of length, and the initial width of the current

sheets (2δ). Spatial distributions for parameters such as num-

ber density n(x), bulk velocity β(x),and their derivation are

also available at Klion et al. 3.

a (auxiliary patch)

f (fine patch)

c (coarse patch)

Parent grid

Absorbing layer

FIG. 2. Sketches of the MR implementation for electromagnetism in

WarpX, where particle charge and current are deposited on the ﬁnest

level ﬁrst, and then recursively interpolated to the coarsest level.

Fields are computed on each grid and patch independently without

boundary interpolation where level 1 patches are terminated by the

absorbing layer (dark green) to prevent the reﬂection of electromag-

netic waves.9

2. Mesh Reﬁnement

To incorporate MR, we build upon the MR strategy in

WarpX further detailed in Fedeli et al. 2. Given the nature

and scope of this paper, we limit our study to a single level

of static(as opposed to adaptive) MR which can be under-

stood through the schematic in Fig. 2. The coarser grid (with

a lower level of reﬁnement) is known as the parent grid (or

level 0 mesh) while the regions with MR are called reﬁne-

ment patches (or the level 1 mesh). The level 1 mesh has

three patches: a ﬁne-resolution patch f, a coarse-resolution

patch cof the same resolution as the underlying level 0 mesh,

and an auxiliary patch of the same resolution as the ﬁne patch.

These independent treatment of these three patches is neces-

sary to reduce artifacts resulting from the constructive inter-

ference with MR.

During particle deposition within the PIC loop, current den-

sity is deposited ﬁrst on fwhere it is then interpolated to

cprior to independently solving Maxwell’s equations on f

and c. By default, ﬁelds are damped at the edges of both

fand cusing a Perfectly Matched Layer (PML)8which is

well suited for particle accelerator simulations with negligible

plasma density at the coarse-ﬁne interface. High plasma den-

sity near the coarse-ﬁne interface in our simulations required

implementing a novel absorbing layer to circumvent the is-

sues observed with the use of a PML. The effect of this new

damping is documented in Sec. V A 3.

After solving Maxwell’s equations on fand c, the solution

is transposed to the auxiliary patch aby the substitution:

F1(a) = F1(f) + I[F0(s)−F1(c)],(1)

where Fis the ﬁeld, Iis the coarse-ﬁne interpolation opera-

tor, Fndenotes the nth level solution of F, and sdenotes the

relevant subset of Fon the parent grid. Finally, to avoid the

artifacts resulting our artiﬁcial damping of the outgoing ﬁelds

from level 1 to level 0, a buffer region (containing the ab-

sorbing boundary region) was added wherein particles gather

SULI Internship Spring 2024 3

from the underlying parent grid instead instead of the auxil-

iary patch.10 This buffer region is referred to as the ﬁeld gather

buffer region, as seen Fig. 1.

3. Initialization

At initialization, the magnetic pressure at the center of the

current sheets is zero. Gas pressure in the current sheet must

compensate to maintain pressure equilibrium, and thus the

current sheet is hotter and denser than the upstream at ini-

tialization. We chose the current sheet number density per

species to be nd=5nb, where nbis the upstream number den-

sity per species. The election-positron plasma momentum is

sampled from a Maxwell-Jüttner distribution at the local tem-

perature with the local bulk velocity.11 The current density

proﬁle of the current sheets follows a tanh2proﬁle in x, cen-

tered on x=±xc.3

To initiate reconnection in a predictable manner, we apply

a one percent sinusoidal perturbation to the vector potential A

given in Klion et al. 3. This reduces the the magnetic pressure

at z=0 in plasma upstream, therefore initiating the tearing-

mode instability which causes the magnetic ﬁeld line colli-

sions leading to magnetic reconnection.

B. Integration of openPMD-API with yt

Prior to this work, yt’s understanding of data with the

openPMD markup was limited to HDF5 ﬁles as the actual ﬁle

handling was accomplished by the h5py Python module. To

expand yt’s capabilities to the other ﬁle formats supported by

the openPMD-api, we modiﬁed the existing openPMD fron-

tend in yt so that data I/O is performed by the openPMD-api.

Our redesign of the software was grounded in ﬁrst recovering

the basic functionality of the existing frontend, and then incor-

porating the grid-based MR tools into the openPMD frontend.

IV. METHODS AND IMPLEMENTATION

A. Magnetic Reconnection Analysis

The novel challenges of incorporating PIC MR for mag-

netic reconnection raised challenges for our analysis of simu-

lation performance. Central to these challenges was the treat-

ment of the course-ﬁne interface when creating visualizations

and examining ﬁeld-based diagnostics. The ﬁeld gather buffer

allows for the accumulation of intense noise and numerical ar-

tifacts within the buffer region without impacting the simula-

tion, so in order to visualize and conduct proper analyses, we

omitted level 1 data within the ﬁeld gather buffer.

Following the work of other PIC analysis and Klion et al. 3,

our metrics for simulation accuracy are 1) visual inspection of

ﬁelds for numerical artifacts, 2) analysis of energy conversion

and conservation, and 3) histograms of particle momentum

(power spectra). We now turn to the speciﬁc methods we de-

veloped for simulations of reconnection with MR.

FIG. 3. The effect of removing the level 1 data from the ﬁeld buffer

gather region is evident in this comparison of the untrimmed (left)

and trimmed (right) version of the Eyﬁeld. The patch edge is denoted

by the black line and the ﬁeld gather buffer edge by the red line.

FIG. 4. A two dimensional Yee grid showing the placement of E and

H (or B) ﬁelds. From Wang et al. 12 . Note the x,yand zaxes in our

simulations correspond to the y,z, and xaxes in this ﬁgure.

a. Field visualizations For our ﬁeld visualizations, the

ﬁeld buffer gather region is masked according the level of the

ﬁeld data. The visual effect of this change is seen in Fig. 3.

b. Energy conservation analysis By default, WarpX av-

erages ﬁeld data to cell centers for the plotﬁle diagnostics

which enables easy portability between a variety of visualiza-

tion frameworks. At runtime however, Maxwell’s equations

are solved at cell nodes on the Yee grid seen in Fig. 4. We es-

tablished a method to use the raw diagnostics on the Yee grid

for our energy calculations.

For the Ez,Bx,Eyﬁelds that are nodal in xon the staggered

Yee grid, special consideration had to be taken at the edge of

SULI Internship Spring 2024 4

the ﬁeld gather buffer region. Because level 1 data nodal in x

is within the inclusive ﬁeld buffer gather region, we omit this

data in favor of level 0 data. Due to the periodic boundary con-

ditions imposed in our simulations, we avoid double-counting

energy values by taking only a single value per ﬁeld per cell,

as all cells share edge values.

The development of these methods for assessing simulation

accuracy enabled the analysis discussed in Section V A.

B. The OpenPMD frontend

To ﬁrst restore the basic functionality of the openPMD fron-

tend, I updated the ﬁle parsing and handling mechanisms to

comply with openPMD-api. Throughout the drafting process,

compatibility tests with HDF5 and ADIOS2 ﬁle formats in

comparison with the legacy h5py-based frontend were con-

ducted using example simulations from WarpX

a. Redeﬁning Data I/O for openPMD-api Excluding the

addition of two trivial classes in the on_demand_imports

and file_handler ﬁles, all development took place within

the openPMD frontend. A ﬁrst pass at functionality required

replacing the key and path-based access to certain parts of

HDF5 data through h5py with the corresponding key and

attribute-based access of data through the openPMD-api.

For I/O performance, the openPMD-api is built with a

I/O logic of opening, registering (possibly many data at

once), ﬂushing, and closing an openPMD-api Series instance.

The implementation of these features is seen in Listing 1.

if extent is not None:

extent += index

if len (record_component.shape) == 3:

registered = record_component[

in d ex [0] : extent [0] , index [1] :

extent [1] , index [2] : ex tent [2]

]

el if len (record_component.shape) == 2:

registered = record_component[

in d ex [0] : extent [0] , index [1] :

extent[1]

]

el if len (record_component.shape) == 1:

regi ste red = r ec or d_ co mp on en t [ index :

extent]

else:

# when we don ’t slice we have to . loa d _chunk

()

regis tere d = r eco rd _ co m po nen t . load _chu n k ()

rec o rd _ co m po n en t . s er i es_ f lus h ()

return np . mu ltiply ( registere d , un it_si )

Listing 1. Within the get_component function of the misc module,

data is registered either by slicing or load_chunk(), and then

ﬂushed with series_flush().

b. Incorporating Mesh Reﬁnement Capabilities To

bring modern analysis techniques to mesh-reﬁned simulation

data, we focused on making our data structures grid-based. By

accessing run-time grids through the available_chunks()

of a particular dataset component, the offset and extent

attributes establish the grids’ locations and sizes.

FIG. 5. Updated energy conservation for three MR simulations

with varying damping proﬁles (solid) are shown in comparison with

their respective energy conversion using cell-centered plotﬁle data

(dashed).

V. RESULTS

A. Mesh Reﬁnement for Reconnection

1. Improved Energy Analysis

As discussed in Section IV A, the improved assessment of

energy conversion/conservation using raw diagnostics shows

improved simulation accuracy as seen in Fig. 5. Similar dif-

ferences in energy conversion/conservation between the initial

and improved methods were seen in all cases tested.

2. MR Parameter Scan

Prior to presenting our best cases for the use of MR, we

brieﬂy introduce the methodological changes we investigated

in this project.

a. Absorbing Boundary vs PML As mentioned in Sec-

tion III A, the default PML implementation in WarpX inad-

equately damped the outgoing(from level 1 to level 0) elec-

tromagnetic waves from our reﬁnement regions due to high

plasma and current density at the course-ﬁne interface. The

default PML only damps the normal components of outgoing

waves, so an updated damping method termed the absorbing

boundary condition (abc) was adopted where both normal and

tangential wave components are damped. The effect of this

change is also seen in Fig. 5, where abc simulations show

better predictability and less susceptibility to the numerical

heating seen to start in the PML simulation at 8000ωc

−1.

b. Dampening Proﬁle and Strength After adoption of

the abc, the results of our investigation of the abc’s damp-

ing strength and proﬁle are shown in Fig.6. Due to the

strange accumulation of energy for our simulation with damp-

SULI Internship Spring 2024 5

FIG. 6. Relative energy conservation for simulations with varying

damping strengths, damping proﬁles, reﬁnement width, and absorb-

ing conditions.

ing strength of 40 with a quadratic damping proﬁle, and the

visual presence of numerical artifacts in the current and mag-

netic ﬁelds, we chose a cubic damping proﬁle with a modest

damping strength of 4 for further optimization.

3. Accuracy Improvement with MR

With our insight about the parameters and damping chal-

lenges with MR discussed in Section V A 2, here we present

the improvements in simulation accuracy seen from apply-

ing MR to a series of parent grids with increasing coarse-

ness. These uniform (no MR) standards are denoted coarse2,

coarse4, and coarse8 run and they will be compared with our

baseline run, a uniform simulation of the maximum resolu-

tion considered in our study. In our study coarseN refers to

the coarsening factor in comparison with baseline (so coarse2

has half the cells of baseline per dimension). This allows for

proximity to the baseline to be the metric for accuracy across

analyses.

To demonstrate the trade-offs in accuracy with perfor-

mance, we apply mesh reﬁnement to coarse2,coarse4, and

coarse8 such that the resolution of the ﬁne patch is equal

to that of baseline. These MR runs are called coarse2 RR2,

coarse4 RR4, and coarse8 RR8 respectively, where RRX refers

to a reﬁnement ratio of Xin each dimensions.

a. The Effect of Parent Grid Resolution To demonstrate

the effect of the coarseness of the parent grid on simulation

accuracy, we ﬁrst hold the total number of particles constant.

We establish uniform standards for the Jyﬁeld in the early

(∼0−2000ωc

−1), middle (∼2000 −4000ωc

−1), and late

(∼4000 −8000ωc

−1) reconnection stages in Fig. 7, where

decreasing grid resolution leads to intense artifacts in the early

and mid-reconnection stages of coarse8 before visually re-

turning to intensities comparable with baseline and coarse4.

The application of MR to coarse2,coarse4, and coarse8

shows visual improvement for Jy(relative to baseline) for all

cases, though we only show the coarse8 and coarse8 RR8

comparison in Fig. 8.

The corresponding analyses for energy conversion and con-

servation are seen in Figs. 9 & 10, where MR shows signiﬁ-

cant improvements for coarse4 and coarse8, but poor perfor-

mance for coarse2.

The effect of MR on particle spectra for the uniform stan-

dards is seen in Fig. 11. In line with the Jyinspection and

energy analysis, most notable is the degree to which coarse8

RR8 improves upon coarse8.

b. Effect of Upstream Particle Density While the pre-

ceding section established the effect of coarsened parent grids

on simulation accuracy, particle density, a primary source of

the computational cost of a PIC simulation, remained con-

stant, greatly limiting the computational beneﬁts of applying

MR to a course parent grid such at coarse8. Therefore a fur-

ther investigation of the limitations on accuracy when decreas-

ing particle density was needed.

Starting with a comparison of baseline,coarse8, and

coarse8 RR8(here labeled 64x64 RP64), Fig. 12 shows the

effect of decreasing particle density outside of the reﬁnement

region on the particle spectra. As expected, decreasing parti-

cle density leads to greater deviations from the baseline, but

primarily this affects the low γregions towards the end of re-

connection.

B. OpenPMD frontend

a. Data IO First the frontend was restored to the func-

tionality of the legacy frontend without updating grids or pro-

viding access to mesh-reﬁnement data.

1. Incorporating Mesh Reﬁnement Capabilities

After the implementation discussed in Section IV B, grids

at multiple levels of reﬁnement are now available for parallel

post-processing workﬂows and other low-level data inspec-

tion. Grid objects now have a functional level attribute and

accurately reﬂect the grids used during runtime for ADIOS2

data. Using data generated from the WarpX ‘plasma mirror’

example with MR, Listing V B 1 shows how to generate Fig.

13, verifying grid agreement between WarpX and yt.

import yt

import num p y as np

gro up se ri es = yt . load ( "/ h ome / hjones / D eskt op /

warpx _sim s / diag_bp / o pe np md_ 00 016 0 .bp ")

sl c = y t . S li ce Pl o t ( ds , ’ z ’, ( ’openPMD’,’ B_y ’) ,

center = [ 0.00004 , 0.00004 ,0])

slc . s et _w id th ((0.000045 , ’ m ’) ,(0.000045 , ’m ’ ))

slc . s et _l og (( ’openPMD’,’B_y ’) , l in th re sh = 0. 9 e2 )

slc . a nn ot at e_ gri ds ()

SULI Internship Spring 2024 6

800

1000

1200

1400

x [

]

930

−

, 0.37

Lz/c

baseline coarse4 coarse8

800

1000

1200

1400

x [

]

2094

−

, 0.83

Lz/c

800

1000

1200

1400

x [

]

3025

−

, 1.21

Lz/c

1000 500 0 500 1000

z [

]

800

1000

1200

1400

x [

]

6981

−

, 2.78

Lz/c

1000 500 0 500 1000

z [

]

1000 500 0 500 1000

z [

]

101

100

10 1

100

101

Jy/

(

)

FIG. 7. A comparison of the normal ycomponent of the current ﬁeld shows signiﬁcant inaccuracy of coarse8 in the early and mid-reconnection

stages (0 to ∼4000ωc−1) before recovering slightly towards the end of reconnection (∼7000ωc−1).

slc . sav e (’ p la sm a_ mir ro r . pn g ’)

Listing 2. Example script for generating a yt SlicePlot with ADIOS2

openPMD data

VI. CONCLUSION AND FUTURE WORK

A. Magnetic Reconnection

This paper has demonstrated our efforts to optimize WarpX

PIC simulations for the particular case of magnetic recon-

nection. Traditional mesh-reﬁnement and adaptive mesh-

reﬁnement schemes are inadequate in capturing the mechanics

of magnetic reconnection due to mass particle transport and

high plasma density.

Here we have shown several cases where MR signiﬁcantly

improves simulation accuracy as determined by visual inspec-

tion of the Jyﬁeld, analyses of the conversion and conserva-

tion of system energy, and the particle spectra. Our uniform

standards coarse4 and coarse8 beneﬁted the most from MR

while maintaining the systems total particle quantity.

In order to fully leverage the efﬁciency potential of MR,

we explored decreasing particle density in the upstream re-

gions, as we hope to focus computational resources close to

the current sheet to study the physics of reconnection. We

have shown that this does reduce simulation accuracy, how-

SULI Internship Spring 2024 7

800

1000

1200

1400

x [

]

930

−

, 0.37

Lz/c

coarse8 coarse8 RR8

800

1000

1200

1400

x [

]

2094

−

, 0.83

Lz/c

800

1000

1200

1400

x [

]

3025

−

, 1.21

Lz/c

1000 500 0 500 1000

z [

]

800

1000

1200

1400

x [

]

6981

−

, 2.78

Lz/c

1000 500 0 500 1000

z [

]

101

100

10 1

100

101

Jy/

(

)

FIG. 8. The application of MR with reﬁnement ratio 8 to coarse8

recovers much of the visual characteristics of the Jyﬁeld for baseline

seen in Fig. 7.

ever primarily towards the end of reconnection in the low γ

regions, where particles are not being accelerated and thus are

not directly related to the particle acceleration of interest to

future studies.

Looking towards the future of MR for PIC plasma sim-

ulations, further optimization is needed before signiﬁcant

speedups from MR are seen in three dimensions. Based on

the primary inﬂuence of particle density on simulation run-

time, a method of the splitting and merging particles across

the ﬁne-course interface effectively increase particle resolu-

tion in the regions with MR. In addition, MR typically shows

improvement for small reﬁned regions, but in our simulations

42% of the domain is reﬁned. Narrowing the reﬁnement re-

gion while maintaining accuracy is an import area of focus.

Additionally, further load balancing and GPU optimization to

reduce communication are directions for future improvement.

B. The OpenPMD frontend

Moving forward, further debugging and testing will be done

prior to our pull request being merged into the yt code base.

Of primary interest is establishing parent and child grid re-

lationships, and optimizing the use of the openPMD-api for

potential parallel and in situ analysis in the future.

FIG. 9. Energy conversion (top) and relative energy conservation

(bottom) for uniform standards show notably poor accuracy for

coarse8 and comparable accuracy for coarse2.

ACKNOWLEDGMENTS

I would like to thank my project mentors and all those who

helped in my work this Spring at LBNL. I am very grateful

for the time and effort that has gone into my project and the

intense learning I was able to experience.

This work was supported in part by the U.S. Department of

Energy, Ofﬁce of Science, Ofﬁce of Workforce Development

for Teachers and Scientists (WDTS) under the Science Under-

graduate Laboratory Internship (SULI) program. Simulations

for this project were performed using the following high per-

formance computing National User Facilities and machines:

Oak Ridge Leadership Computing Facility (OLCF): Sum-

mit, National Energy Research Scientiﬁc Computing Center

(NERSC): Perlmutter. The magnetic reconnection code was

created using the development branch of WarpX built from

the AMReX framework, which is developed and maintained

by the Center for Computational Sciences at Lawrence Berke-

ley National Laboratory. We used the latest development of

WarpX with additional features available at Jambunathan 13 .

Post processing was executed in Jupyter Notebook using the

SULI Internship Spring 2024 8

FIG. 10. The application of MR to the uniform standards shows sig-

niﬁcant improvement on coarse4 and coarse8, but poor performance

for on coarse2.

python libraries numpy, matplotlib, and yt. Simulated cases

were visualized using yt and Paraview.

1W. Zhang, A. Almgren, V. Beckner, J. Bell, J. Blaschke, C. Chan, M. Day,

B. Friesen, K. Gott, D. Graves, M. Katz, A. Myers, T. Nguyen, A. Nonaka,

M. Rosso, S. Williams, and M. Zingale, “AMReX: a framework for block-

structured adaptive mesh reﬁnement,” Journal of Open Source Software 4,

1370 (2019).

2L. Fedeli, A. Huebl, F. Boillod-Cerneux, T. Clark, K. Gott, C. Hillairet,

S. Jaure, A. Leblanc, R. Lehe, A. Myers, C. Piechurski, M. Sato, N. Zaim,

W. Zhang, J.-L. Vay, and H. Vincenti, “Pushing the frontier in the de-

sign of laser-based electron accelerators with groundbreaking mesh-reﬁned

particle-in-cell simulations on exascale-class supercomputers,” in SC22:

International Conference for High Performance Computing, Networking,

Storage and Analysis (2022) pp. 1–12.

3H. Klion, R. Jambunathan, M. E. Rowan, E. Yang, D. Willcox, J.-L. Vay,

R. Lehe, A. Myers, A. Huebl, and W. Zhang, “Particle-in-cell simulations

of relativistic magnetic reconnection with advanced maxwell solver algo-

rithms,” The Astrophysical Journal 952, 8 (2023).

4“yt Overview; The yt Project 4.3.0 documentation — yt-project.org,”

https://yt-project.org/doc/index.html (2014), [Accessed 02-

02-2024].

5A. H. et al., “openPMD — openpmd.org,” https://www.openpmd.org/

#/start, [Accessed 22-02-2024].

FIG. 11. Particle Spectra for the uniform standards (dashed) are

shown in comparison with the corresponding application of MR (dot-

ted).

6“openPMD C++ & Python API &x2014; openPMD-api 0.15.2 doc-

umentation — openpmd-api.readthedocs.io,” https://openpmd-api.

readthedocs.io/en/0.15.2/index.html, [Accessed 02-02-2024].

7E. G. Harris, “On a plasma sheath separating regions of oppositely directed

magnetic ﬁeld,” Il Nuovo Cimento 23, 115–121 (1962).

8J.-L. Vay, “Asymmetric perfectly matched layer for the absorption of

waves,” Journal of Computational Physics 183, 367–399 (2002).

9“Mesh reﬁnement,” https://warpx.readthedocs.io/en/24.01/

theory/amr.html (2024), [Accessed 17-04-2024].

10J.-L. Vay, D. P. Grote, R. H. Cohen, and A. Friedman, “Novel methods in

the particle-in-cell accelerator code-framework warp,” Computational Sci-

SULI Internship Spring 2024 9

FIG. 12. The effect of reducing particle density outside of the reﬁne-

ment patch, where n x n RP X represents the a run with nparticles

per cell per dimension outside of the reﬁnement patch and Xtotal

particles per cell in the reﬁned region. Here 64 x 64 RP64 = coarse8

RR8, as we are examining this effect for our case where MR shows

the most improvement.

FIG. 13. Demonstrated MR capabilities for the newly redesigned

openPMD frontend within yt. In this simulation with only one level

of reﬁnement, the white lines are the edges of the level 0 grids and

the black lines are the edges of the single level 1 grid.

ence Discovery 5, 014019 (2012).

11S. Zenitani, “Loading relativistic Maxwell distributions in particle simu-

lations,” Physics of Plasmas 22, 042116 (2015), arXiv:1504.03910 [astro-

ph.HE].

12Y. Wang, Y. Cheng, X.-H. Wang, S. Yang, and Z. Chen, “An sbp-sat fdtd

subgridding method using staggered yee’s grids without modifying ﬁeld

components for tm analysis,” IEEE Transactions on Microwave Theory and

Techniques 71, 579–592 (2023).

13R. Jambunathan, https://github.com/RevathiJambunathan/WarpX.

14DOE, “DOE Explains...Exascale Computing — energy.gov,” https:

//www.energy.gov/science/doe-explainsexascale-computing,

[Accessed 02-02-2024].

15H. Jones, https://github.com/yt-project/yt/pull/4848.

16“Acm gordon bell prize,” https://awards.acm.org/bell (2024), [Ac-

cessed 17-04-2024].

17J.-L. Vay, “An extended fdtd scheme for the wave equation: Application to

multiscale electromagnetic simulation,” Journal of Computational Physics

- J COMPUT PHYS 167, 72–98 (2001).