1.1486 is my latest center point prediction...

I hope my hedging this month pays off better than the last....

The ensembles appear to have shifted downwards the last two runs for t2m around the 24th..

Compare the anomaly from 2 days ago to today's estimate

The exercise to discern why it is colder in today's run (especially around the 24th) is left to the reader.

center point prediction now of 1.134 C

End of month dropped again (judging visually, last few days dropped by about 0.08 C).

EPS showing a different scenario for NW US from the 26th onwards (jet stream meandering further south with a (originally mid-level low merged from two other mid level lows over Alaska) low that drops now further south than the other models (AIFS also shows this scenario).

(Eastern) Antarctica got cooler as well.

Yesterday's run for last day of month:

Today's run:

Based off of June 1, 00-06Z run:

Another plataeu mid month seems indicated by medium range superensemble and shape of trend for the extended range subseasonal model.

The subseasonal continues to remain higher, although the medium range has shifted downwards towards the end of its forecast in the two runs prior.

~

For my future reference:

I spent the last few hours finally tuning the weighting for the statistical model I've been using for reference in the first 10 days for the last year (arimax, 1-harmonic) against the dynamic superensemble+Prophet now that I have a year of data.

For future reference (on the first of the month), the variance for the statistical is ~ 3 times the dynamic+super-ensemble

monthly_predictive_var, monthly_predictive_var_stat, monthly_predictive_var_weighted

(0.0044, 0.0127, 0.0075) respectively

Since I'm weighting by 1/MSE by lead time days (accounting for days left in prediction for month), the weights end up for

weight_for_statistical, weight_for_super_and_prophet

0.38, 0.62 respectively.

This was a bit surprising against my intuition how much weight is given to the super ensemble+Prophet early on in the month, which shows I have been underweighting it in the first 10 days in the past.

I've also perhaps been slightly underestimating the variance at this stage (nothing above 20% is indicated with this much variance at this stage in this weighted model; in the past I think I went up to 25%?, and also a bit too harshly bet down far away bins).

~

Example chart from June 1 using the statistical (ARIMAX) model (not Prophet):

Toggling between the two, the super ensemble is clearly above the statistical one in the medium range, and only slightly above for the comparable Prophet portion beyond that.

~

So other than the ad-hoc shift (which is large) by using the recent errors for each of the three ERA5->GISTEMP models, everything else is as objective as I can make it now.

For the moment I get a (after all adjustments) a point prediction of ~ 1.21 C for June:

Point estimate (mix) adjusted by prediction error mean: 1.2067

I've also changed my betting strategy based on these probabilities since last month or so to be more algorithmic in the calculation of bet sizes but its still not ideal.

Tiny dip and a rise is now clear at end of medium range. EC46 from couple days ago still about 0.14C below end of mid-range, but shifted to it continues to follow Prophet extende from medium range and close to the ARIMAX estimator from ERA5 also.

Since I've been weighting the statistical (1 harmonic) ARIMAX from the beginning of the ERA5 period in the data I use (unfortunately the code I have only adjusts the actual data for the month of interest) I now output the combined for the medium range and where it overlaps with prophet (the stat arimax doesn't go out as far (45 days) as Prophet, months into the future).

Anyway for the the medium range you can see the statistical model's effect on the result where it raises the temps in the middle of the medium range (where there is a dip below what you would expect) and lowering it at the end of the medium range (where there is a rise on tau=D+16, although some of that is from GEFS+GEPS biases where EPS is not available then).

I still am not using EC46 for after the medium range in terms of predictions (its just for reference).

You can also observe the widened (weighted) CI in the medium range as a result.

Superensemble from same runs today (through 06Z):

super ensemble + prophet extended from it (with EC46 as reference):

(super ensemble + prophet extended from it) weighted against statistical arimax (1 harm, from end of ERA5)

@parhizj yikes.

@ScottSupak If looking at EC46, don't look too much into the absolute numbers (since it is shift is ad-hoc) just the trend, which shows for the last 2-3 runs, with the mean case there will be roughly a plataeu on average going into July for the first week or so and then a trend upwards on average.

@parhizj looks like the tail risk is mostly on the upside, yeah?

@ScottSupak All of my models assume a normal distribution on point predictions (I only try to account for the variance of everything -- I don't even bother trying to correct the skew/kurtosis).

I think those betting the lowest bin down are just betting it won't be anomalously cooler than April, which is a fair bet all things considered given that preliminary (not well debiased yet by what little empirical data I gather) SST forecasts for daily (anomalized) versions of ONI/RONI based off of ERA5/EC46 data that I process are higher for June than April so that should be good reason to assume it will also carry over to the LOTI value.

Also CPC officially put out an El Nino advisory today: https://www.cpc.ncep.noaa.gov/products/analysis_monitoring/enso_advisory/ensodisc.shtml

~

On other randomness in case it may help anyone... I wasted hours try to setup a pipeline to download the rest of the ERA5 data faster from other sources (Earth Data Hub) only to find out that the data is quantized (0.0156 K spacing) to an unacceptable level for my future purposes (cds being from the daily post-processed dataset which are about 2MB / day, and the EDH which are 0.8MB) ...

>>> cds_vals = cds["t2m"].values.squeeze()

>>> edh_vals = edh["t2m"].values.squeeze()

>>> # look at the least significant bits by checking how many unique values there are

>>> print("cds unique values:", np.unique(cds_vals).shape[0])

cds unique values: 529432

>>> print("edh unique values:", np.unique(edh_vals).shape[0])

edh unique values: 3779

>>> # and the spacing between sorted unique values

>>> cds_sorted = np.sort(np.unique(cds_vals))

>>> edh_sorted = np.sort(np.unique(edh_vals))

>>> print("cds min spacing:", np.diff(cds_sorted).min())

cds min spacing: 6.1035156e-05

>>> print("edh min spacing:", np.diff(edh_sorted).min())

edh min spacing: 0.015625

>>> print("cds median spacing:", np.median(np.diff(cds_sorted)))

cds median spacing: 9.1552734e-05

>>> print("edh median spacing:", np.median(np.diff(edh_sorted)))

edh median spacing: 0.015625

Climate Prediction Center: ENSO Diagnostic Discussion

Currently getting around 1.16 C LOTI for June with updated setup, but its been shifting up to 0.02C a few times in the last few days around there...

~

(Ignore the July curve below -- EC46 looks shifted far too high (+0.31C from its calibrated baseline) from GEFS/GEPS at the end of the range -- same for statistical) (need to revisit how to use these dated subseasonal forecasts and update them with the more updated medium-range models since there should be a better way than this ad-hoc method I came up with)

The super ensemble seems to have done fairly poorly as of late after 7d, with it being biased warm...

~ Some experiments: ~~

My new experimental (daily) SST dashboard that includes EC46 has shifted upwards now going into June and July, compared to the runs last week (still below 1997 -- closer to 1987, 2015 further out). Large grain of salt needed for this project since I just started it and despite having a few samples last week's validation for this reforecast-type calibration has been very poor (biased too cold (~ -0.2C, up to -0.3C is terrible): forecasts calibrated too warm for the tropics and too cold in NINO3.4 region) so if the trend continues there will continue to be a cold bias in my calibration (this may be due to using 1deg gridded data or for some other reason -- perhaps I shouldn't calibrate the regions separately and calculate it from that). I'm still new to this type of subseasonal calibration using reforecasts so more work looks to be needed... this might be also just that its looking to be a very strong/super El Nino and the recalibration years (2006-2025) are poor priors in terms of biases.

the two white diagonals are missing forecasts... (haven't coded it to blacken it out yet)

Z-mean square is terrible also (too large spread) despite trying to carefully fix it using splines. Unfortunately since I don't have NRT SST (the ERA5 SSTs are lagged 5 days) that could be cross-calibrated I can't figure out how to use timely bias correction for the dated model (2 days old)

Compared to the offical Seasonal plumes from ECMWF that are around +1.1C, my estimate is a bit less than 0.9 C for June. Worse is for July (which isn't complete yet), which ECMWF has around 1.5-1.6C or so, while I have (extrapolating a bit for the last 2 days) I have 1.16 C (about 0.4C colder). Could be very well my calibration is that bad given I've seen up to -0.3C 10 days out (and this is 46 days out!).

~~

I've experimented in the last couple days, creating DOY climatologies (365d ones from a 366d with Feb 29 redistributed and then removed, smoothed with a gaussian, 7d window (for ERA5 and super ensemble analysis) and one with a 5d window for pure medium-range forecasts) to extend the delta analog method (for GISTEMP-LOTI) usability to month-to-date anomalies (for prior to the 15th of the month for instance).

Since I only have disk space for going back to 1980 for this much data (the daily gridded data and the climatologies together take up a lot) its hard to make an apple-apple comparison with the previous (monthly delta analog) one, as it is slightly outperforming the DOY one (the larger dataset gives it too large of an advantage).

At the very least I will be able to produce some useful MTD anomaly diagnostic plots (monthly and doy below already have the full superensemble coverage so its not obvious as the difference is subtle).

Russia and the Antarctic Peninsula will be anomalously warm again this month. With the Antarctic heatwave making news:

Record winter temperatures in Antarctic raise fears over speed of climate breakdown

~

As for diagnostics for the climatologies its worth noting that the differences between the monthly anomaly (which is not smoothed) and the doy anomaly are much more muted for the months when the temperatures are not changing fast, i.e. that's why Jan-Feb and July-Aug have the lowest deltas when comparing the doy ("new") and raw monthly ("old") climatologies:

~

In addition to above work, for the loti analog method and the delta analog method (but not yet the original/old GISTEMP-ERA5 model) I've come up with a more elaborate generalized variance method that seems to be better calibrated (adjusting for everything I can think of now).

Also much better (lagged) tracking of the performance for the final ad-hoc debiasing step using the last 12 months or so for which I have release-date data that lets me know how much of each of the models is benefitting from this step and accordingly how to weight (still by inv. mse, but taking into this account now) -- its quite a bit more complex, but more automated and objective now, and so better justified than just hoping the trailing ad-hoc bias correction works equally well for all the models (now we have some objective way of determining the weights). The contributions/effectivenss will probably vary by month, since the loti analog and the delta analog methods (monthly and doy) use different per month models and the loti analog method also tunes each monthly model on the cutoff date (>= 1940, 1950, etc).

Still left to do is getting the same metrics from the naive old/original method so they are all weighted properly (at the moment the variance is naive and not a forecast like the others -- the seasonality for each model isn't well calibrated). Beyond this there isn't much more to do with this type of statistical ensemble analysis with pure t2m data...

~~~~~

Random pro-tips for the day for grib processing:

1) Instead of running grb.latlons() (forpygrib) on every file, if expecting an llgrid, cache on the header keys instead (this is a ~ 10x speedup, as the mesh creation is SLOW) and just get the latlons once:

(

int(grb.Ni),

int(grb.Nj),

float(grb.latitudeOfFirstGridPointInDegrees),

float(grb.latitudeOfLastGridPointInDegrees),

float(grb.longitudeOfFirstGridPointInDegrees),

float(grb.longitudeOfLastGridPointInDegrees)

)

2) if regridding alot of files (ensemble members), always aggregate in input space first (duh....)

3) if abundant disk space (certainly not me!) and have free CPU ahead of other main processing (i.e. process as as you download data steps), decompress grib files ahead of time, like using grib_to_netcdf.

~

Record winter temperatures in Antarctic raise fears over speed of climate breakdown

Temperatures above 15C ‘very strange’ say scientists, as snow melts and rain falls on glaciers in usually frozen region