Accessing evolutionary Coeval data

When you run run_coeval with non-saturated spin temperature fluctuations (or various other options, such as inhomogeneous recombinations), or you run run_lightcone, many coeval simulation cubes at higher redshifts are computed, to generate the evolution up to the redshift you requested (or in the case of lightcones, to be interpolated over). By default, with write=True, all these boxes are saved to the cache (or a folder of your choice). What is the best way to access that data?

Looking at the output datafiles themselves (a bunch of .h5 files) is fairly confusing – the files are saved with names based on inscrutable hashes. You can use the builtin command-line functionality 21cmfast query to identify what’s what, but that can be a bit clunky. You could also just run run_coeval at the redshift you care about, with the same input parameters – this will just return the cached object. But this is also a bit clunky, and requires you to know the redshifts that were calculated during the evolution.

Never fear though – both the Coeval and LightCone objects have a get_cached_data() method that will find these files for you! Let’s see how it works.

[1]:
%matplotlib inline
import matplotlib.pyplot as plt
import os

import py21cmfast as p21c

# For plotting the cubes, we use the plotting submodule:
from py21cmfast import plotting

# For interacting with the cache
from py21cmfast import cache_tools

import h5py
import numpy as np
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_cfg.py:57: UserWarning: Your configuration file is out of date. Updating...
  warnings.warn(
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_cfg.py:41: UserWarning: Your configuration file is out of date. Updating...
  warnings.warn("Your configuration file is out of date. Updating...")
[2]:
print(f"Using 21cmFAST version {p21c.__version__}")
Using 21cmFAST version 3.3.2.dev97+g184d592.d20231222

We’re going to clear the cache, so that this notebook produces the same output every time. You probably shouldn’t do this yourself (unless your cache is getting too big!).

We also set the default output directory to _cache/ to keep all of the outputs of this notebook in one place:

[3]:
if not os.path.exists('_cache'):
    os.mkdir('_cache')

p21c.config['direc'] = '_cache'
cache_tools.clear_cache(direc="_cache")

Create an example box

We first run a coeval box and lightcone to have something to read later. It’s a really small and unrealistic simulation (the point here is to show how to use the cached files, not to get something that looks pretty!):

[4]:
coeval = p21c.run_coeval(
    redshift = 25.0,
    user_params = {"HII_DIM": 30, "BOX_LEN": 45},
    flag_options={"USE_TS_FLUCT": True},
    random_seed=1978,
    write=True
)

lightcone = p21c.run_lightcone(
    redshift = 25.0,
    max_redshift = 35.0,
    user_params = {"HII_DIM": 30, "BOX_LEN": 45},
    flag_options={"USE_TS_FLUCT": True},
    random_seed=1978,
    lightcone_quantities=("brightness_temp", 'density'),
    global_quantities=("brightness_temp", 'density', 'xH_box'),
    write=True
)
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/inputs.py:514: UserWarning: The USE_INTERPOLATION_TABLES setting has changed in v3.1.2 to be default True. You can likely ignore this warning, but if you relied onhaving USE_INTERPOLATION_TABLES=False by *default*, please set it explicitly. To silence this warning, set it explicitly to True. Thiswarning will be removed in v4.
  warnings.warn(
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vx
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vy
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vz
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vx_2LPT
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vy_2LPT
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
/home/steven/work/21cmfast/21cmfast/src/py21cmfast/_utils.py:811: UserWarning: Trying to remove array that isn't yet created: hires_vz_2LPT
  warnings.warn(f"Trying to remove array that isn't yet created: {k}")
2023-12-28 11:57:01 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.493237e+01
2023-12-28 11:57:03 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.451287e+01
2023-12-28 11:57:05 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.355277e+01
2023-12-28 11:57:07 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.262872e+01
2023-12-28 11:57:09 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.173945e+01
2023-12-28 11:57:12 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.088374e+01
2023-12-28 11:57:14 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.006038e+01
2023-12-28 11:57:16 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.926822e+01
2023-12-28 11:57:18 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.850615e+01
2023-12-28 11:57:20 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.777308e+01
2023-12-28 11:57:22 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.706797e+01
2023-12-28 11:57:25 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.638980e+01
2023-12-28 11:57:27 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.573761e+01
2023-12-28 11:57:29 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.511044e+01
2023-12-28 11:57:32 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.450738e+01
2023-12-28 11:57:34 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.392755e+01
2023-12-28 11:57:36 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 1.337012e+01
2023-12-28 11:57:39 | INFO    | SpinTemperatureBox.c | ComputeTsBox:549 [pid=10384/thr=0] | previous_spin_temp: 2.548283e+01

Accessing cached data from Coeval

The function to use here is get_cached_data. To use it, pass the (approximate) redshift that you want, and the kind of output that you want (init, perturb_field, ionized_box, spin_temp or brightness_temp). The method will grab the closest redshift to the one you pass that was actually calculated in the evolution of the given box. By default, the returned object will be empty, and you can read in the data manually. To read in the data automatically, set load_data=True. Let’s see it in action:

[5]:
brightness_temp_z25 = coeval.get_cached_data(redshift=25.1, kind='brightness_temp', load_data=True)

The output is an OutputStruct of whatever kind you requested:

[6]:
print(brightness_temp_z25)
BrightnessTemp(UserParams(BOX_LEN:45, DIM:90, FAST_FCOLL_TABLES:False, HII_DIM:30, HMF:1, KEEP_3D_VELOCITIES:False, MINIMIZE_MEMORY:False, NON_CUBIC_FACTOR:1, NO_RNG:False, N_THREADS:1, PERTURB_ON_HIGH_RES:False, POWER_SPECTRUM:0, USE_2LPT:True, USE_FFTW_WISDOM:False, USE_INTERPOLATION_TABLES:True, USE_RELATIVE_VELOCITIES:False);
        CosmoParams(OMb:0.04897, OMm:0.3096, POWER_INDEX:0.9665, SIGMA_8:0.8102, hlittle:0.6766);
        random_seed:1978;
        redshift:25.0;
        FlagOptions(APPLY_RSDS:True, FIX_VCB_AVG:False, INHOMO_RECO:False, M_MIN_in_Mass:False, PHOTON_CONS:False, SUBCELL_RSD:False, USE_CMB_HEATING:True, USE_HALO_FIELD:False, USE_LYA_HEATING:True, USE_MASS_DEPENDENT_ZETA:False, USE_MINI_HALOS:False, USE_TS_FLUCT:True);
        AstroParams(ALPHA_ESC:-0.5, ALPHA_STAR:0.5, ALPHA_STAR_MINI:0.5, A_LW:2, A_VCB:1, BETA_LW:0.6, BETA_VCB:1.8, F_ESC10:0.1, F_ESC7_MINI:0.01, F_H2_SHIELD:0, F_STAR10:0.05012, F_STAR7_MINI:0.01, HII_EFF_FACTOR:30, ION_Tvir_MIN:50000, L_X:1e+40, L_X_MINI:1e+40, M_TURN:5.012e+08, NU_X_THRESH:500, N_RSD_STEPS:20, R_BUBBLE_MAX:15, X_RAY_SPEC_INDEX:1, X_RAY_Tvir_MIN:50000, t_STAR:0.5))

We can proceed to plot the brightness temperature that we read from cache – and since we only calculated up to \(z=25\) for the Coeval, the cache should be pointing to the same brightness temperature:

[7]:
fig, ax = plt.subplots(1,2, figsize=(10,4))
plotting.coeval_sliceplot(coeval, ax=ax[0], fig=fig)
plotting.coeval_sliceplot(brightness_temp_z25, ax=ax[1], fig=fig)
plt.tight_layout()
../_images/tutorials_gather_data_15_0.png

Of course, we could have pulled the cached data from a higher redshift:

[8]:
st_z35 = coeval.get_cached_data(redshift=35.0, kind='spin_temp', load_data=True)
st_z25 = coeval.get_cached_data(redshift=25.0, kind='spin_temp', load_data=True)

fig, ax = plt.subplots(1,2, figsize=(10,4))
plotting.coeval_sliceplot(st_z35, ax=ax[0], kind='Ts_box')
ax[0].set_title("z=35")
plotting.coeval_sliceplot(st_z25, ax=ax[1], kind='Ts_box')
ax[1].set_title("z=25")
plt.tight_layout();
../_images/tutorials_gather_data_17_0.png

Here we see evolution in the spin temperature field. Note that we can’t look at the brightness temperature field at \(z=25\) because it was never saved as part of the evolution of the coeval box (only the ionization field and spin temperature field are saved). This is different in run_lightcone – all boxes are cached at every evaluated redshift for the lightcone.

Accessing Cached Data from a Lightcone

The process is essentially exactly the same for a lightcone:

[9]:
brightness_temp_z25 = lightcone.get_cached_data(redshift=25.0, kind='brightness_temp', load_data=True)
brightness_temp_z35 = lightcone.get_cached_data(redshift=35.0, kind='brightness_temp', load_data=True)
[10]:
fig, ax = plt.subplots(1,2, figsize=(10,4))
plotting.coeval_sliceplot(brightness_temp_z35, ax=ax[0])
ax[0].set_title("z=35")
plotting.coeval_sliceplot(brightness_temp_z25, ax=ax[1])
ax[1].set_title("z=25")
plt.tight_layout();
../_images/tutorials_gather_data_22_0.png

Note that this time, we are able to access the brightness temperature data from the cache, because lightcones save it all.

In fact, we could go further than this, and plot all the evaluated brightness temperatures. We can get the list of redshifts evaluated from the lightcone directly as node_redshifts:

[11]:
fig, ax = plt.subplots(len(lightcone.node_redshifts)//3, 3, figsize=(14,4*len(lightcone.node_redshifts)//3))

for i, z in enumerate(lightcone.node_redshifts):
    Tbz = lightcone.get_cached_data(redshift=z, kind='brightness_temp', load_data=True)
    plotting.coeval_sliceplot(Tbz, ax=ax.flatten()[i])
    ax.flatten()[i].set_title(f"z={z:.2f}")
plt.tight_layout()
../_images/tutorials_gather_data_25_0.png

To illustrate another way of accessing these cached boxes, we can simply run run_coeval at a particular redshift, with the same input arguments. Let’s choose about \(z\sim30\):

[12]:
z30 = lightcone.node_redshifts[8] # the closest to 30

# Run the coeval. This will just find the box in cache, since it has exactly the same parameters.
coeval_z30 = p21c.run_coeval(
    redshift = z30,
    user_params = {"HII_DIM": 30, "BOX_LEN": 45},
    flag_options={"USE_TS_FLUCT": True},
    random_seed=1978,
    write=False
)

# Compare that to what we get from the lightcone cache
lc_z30 = lightcone.get_cached_data(redshift=30, kind='brightness_temp', load_data=True)

fig, ax = plt.subplots(1,2, figsize=(10,4))
plotting.coeval_sliceplot(coeval_z30, ax=ax[0])
plotting.coeval_sliceplot(lc_z30, ax=ax[1])
plt.tight_layout()
../_images/tutorials_gather_data_27_0.png

These are exactly the same. Note that in fact, the data comes from exactly the same file (the lightcone ‘cache’ is just a pointer to the standard file cache, just easier to access).

Gathering cached data into one file

As we’ve already noted, when you use get_cached_data, the Coeval or Lightcone objects just know how to access a specific cached file – usually found in your configured cache directory. This can be a little brittle – if the cache files are removed, the Coeval object won’t be able to access that data any more. To protect against this, you can gather all the evolutionary data into a single file. It’s typically easiest if you first save the lightcone:

[13]:
fname = lightcone.save(fname='lightcone.h5', direc='_cache')

Then, you can gather the cached data into that file:

[14]:
lightcone.gather(
    fname=fname,
    kinds=("brightness_temp", "init"),
    direc='_cache',
);

The above gathered the brightness temperature and initial conditions data, and saved it into the file _cache/lightcone.h5. The kinds can be any of those allowed in get_cached_data. You can also call it again to gather more data:

[15]:
lightcone.gather(fname=fname, kinds=('ionized_box',));

The original cached data files are left on-disk, which duplicates data. If you’re sure you want to remove them, pass clean=True to gather.

The structure of the gathered data in the file is the following:

[16]:
with h5py.File(fname, "r") as fl:
    print(fl['cache'].keys())
    print(fl['cache']['brightness_temp'].keys())
    print(fl['cache']['brightness_temp']['z25.00'].keys())

    tb_25 = fl['cache']['brightness_temp']['z25.00']['brightness_temp'][...]
<KeysViewHDF5 ['brightness_temp', 'init', 'ionized_box']>
<KeysViewHDF5 ['z25.00', 'z25.52', 'z26.05', 'z26.59', 'z27.14', 'z27.71', 'z28.28', 'z28.87', 'z29.46', 'z30.07', 'z30.69', 'z31.33', 'z31.97', 'z32.63', 'z33.31', 'z33.99', 'z34.69', 'z35.41']>
<KeysViewHDF5 ['brightness_temp']>

We’ve read in the brightness temperature at the final redshift. We can compare this to the coeval box directly evaluated there:

[17]:
T21minplot=-10.0
T21maxplot=10.0

fig, ax = plt.subplots(1,2, figsize=(10,4))
plotting.coeval_sliceplot(
    coeval,
    ax=ax[0], fig=fig, kind="brightness_temp", slice_axis=0,
    vmin=T21minplot, vmax=T21maxplot, cmap = "magma"
)
ax[1].imshow(
    tb_25.T[:, :, 0],
    origin="lower", vmin=T21minplot, vmax=T21maxplot, cmap = "magma"
)
plt.tight_layout();
../_images/tutorials_gather_data_40_0.png

Again, they are equivalent.