Interpolating data for higher-resolution pathway studies

[LRydin]

Issue first opened as a PR in PyPSA-Eur's PR #1986, now rightfully closed. For congruence, I'll copy-paste here the PR as was, with the caveat that the proposed were a work-around for PyPSA-Eur, not technology-data.

I've been toying around with the possibility of running year-by-year miopic optimisation. Shortly, I found that the only two extant obstacles were:

• Missing yearly biomass potentials
• Missing yearly technology costs

Thus, I've changed two things in this PR. First, I've created an elementary script that linearly interpolates the biomass potentials. Secondly, I've changed the retrieve.smk rule for the costs data from technology-data's `output'.

I'd like to ask if it possible to either:

• Generate a year-by-year technology data for myoptic optimisation in PyPSA-Eur that would like to investigate finer pathways resolutions, or
• Generate a script that locally interpolates data, thus retaining the existent structure in technology-data and interpolate the data locally for each user.

[LRydin]

Correction: this is marked as a bug, but it is a feature request. For some reason the feature request option is not working in GitHub.

[euronion]

Thanks for the suggestion! With the refactored technology-data this will be much simpler. However we do not yet have a delivery date, probably in the orders of months until we get it shipped.

If you need it before, I see two options:
a) Set up a script locally for your PyPSA-Eur fork (I know, this is cycling back to PyPSA-Eur, but would only be a change in your fork - we wouldn't do it in the main repo)
b) We could generate files for all years in between 2020 and 2050, instead of only on a 5-yearly basis. I'd be open to such a change.

[LRydin]

Thanks @euronion. Regarding your two options, for anyone also interested in this, this is not a difficult task in general, that is, interpolating the data in the technology-data. pandas can easily interpolate the data between years.

@euronion, on another note, I think it could make sense to pay some attention to the likely inherent power-law-like scaling of technology costs. If we simply linearly interpolate between each year, we don't get an accurate representation. Thus, having all CSVs for all years available for download from technology-data might make sense (so, your option b)) -- but this will substantially increase the repository size.

[euronion]

Thanks @euronion. Regarding your two options, for anyone also interested in this, this is not a difficult task in general, that is, interpolating the data in the technology-data. pandas can easily interpolate the data between years.

Let's have people go with option a) then and not spent time on it here :)

@euronion, on another note, I think it could make sense to pay some attention to the likely inherent power-law-like scaling of technology costs. If we simply linearly interpolate between each year, we don't get an accurate representation. Thus, having all CSVs for all years available for download from technology-data might make sense (so, your option b)) -- but this will substantially increase the repository size.

We don't do any scaling. The data we provide is based on external scenarios/studies. If the input data has different data for different years, hopefully accounting for scaling, then interpolation would just be gap filling. If the external data does not contain or forsee any developments, we just keep the values constant. E.g. we have a couple of entries with only a single year as input which we apply as constant to all other years.