Cleanup of standard TSMP test cases #167

Activity

1. DCaviedesV

   added

   datasetsAnything related to public datasets, model datasets, validation, etcAnything related to public datasets, model datasets, validation, etc

   test setupsConcerning supported test setupsConcerning supported test setups

   on Mar 3, 2023
2. DCaviedesV

   assigned

   chartick

   on Mar 3, 2023
3. chartick commented on Mar 7, 2023

   

   chartick

   on Mar 7, 2023

   Contributor

   More actions

   All folders in /bldsva/setups/

   • bonnRadar: Deprecated. You can mostly set it up, but the surface files of CLM refer to a path on a Bonn machine, and nobody in our group used the domain before.
   • common: Seems a bit misplaced. Contains functions for the common setup script, but also scripts to create weight files for ICON-CLM.
   • cordex: Running but maybe needs an update in the static files?
   • germany: Crash during runtime. This is a setup for ICON over Germany from @s-poll. It is still mentioned in the README, but the instructions mention git checkout TSMP_iconcoup which is not available anymore. It is possible to create a setup with TSMP, but there seems to be environment issues for this ICON version.
   • icon-ccs: Not running. Setup for ICON with CLM3.5 and ParFlow. The static files for the setup still exist on the Jülich supercomputers, but the included generation of remap files doesn't work. The reason is a strange way of loading the environment, which fails to load NCO and CDO.
   • ideal1200600/ideal24001200/ideal300150/ideal600300/idealRTD: Working.
   • idealLES: Not working, only fragments exists because this was only running on CLUMA2.
   • multi-scale: Deprecated. Only on JURECA with missing data for the slopes in ParFlow.
   • nrw-icon: Not working. Seems to be an adaption of icon-ccs for nrw.
   • nrw: Working. Probably there is also room for improvement similar to cordex but with less priority.
   • restart: Deprecated. Just contains a shell-script to adapt the restarts paths of a fully coupled run, like it is done is some workflows. The name restart functionality implies more than there actually is. In any case, not really part of a setup and confusing for new users.
   • rur: Deprecated. The location of the needed static files is unknown. The setup also contains paths to a compute time project that does not exist anymore. Maybe the files still exist somewhere.
   • scalingStudy: Deprecated. I think that is a derivate of rur referencing the same compute time project. Especially confusing because the name implies that this is a setup to perform for new compute time projects.
   • seebreeze: Depracted. Static data missing for COSMO and CLM.
   • smresponse: Depracted. Static data missing for COSMO and CLM.

   Nrw, ideal-cases and cordex are working. The ICON-setups could potentially be fixed. All the other stuff is not useful anymore in my opinion.
4. DCaviedesV commented on Mar 7, 2023

   

   DCaviedesV

   on Mar 7, 2023

   ContributorAuthor

   More actions

   Great summary. The conclusion seems reasonable to me. I would guess @s-poll may have some opinions on idealLES, nrw-icon and germany.

   I also wonder what multi-scale is. @s-poll maybe you know?

   Before fully deprecating any of these cases due to missing static files, I'd ask @spoll @niklaswr @kgoergen @skollet for hope on finding these static files. I would also ask: is it of interest to maintain these as supported cases? What I mean is that these supported cases are meant to be examples which can be easily simulated, but this is not the place to maintain setups of interest for production.

   @chartick, if I understood correctly the content in common is some collection of functions for setup_tsmp.ksh. Could we not simply move these functions into setup_tsmp.ksh?
5. chartick commented on Mar 7, 2023

   

   chartick

   on Mar 7, 2023

   Contributor

   More actions

   The ICON-setups are germany, icon-ccs and nrw-icon.

   multi-scale seems to be a scaling study with CLM-ParFlow.

   Yes, with the clean-up of the setup, we should be able to move it.
6. kgoergen commented on Mar 7, 2023

   

   kgoergen

   on Mar 7, 2023

   Contributor

   More actions

   We have now a split between building the model system and providing experiments which consist of setups and configurations, based on a common workflow engine (aka "templates", etc.). In the future it is intended to also make these experiments (e.g., EURO-CORDEX CMIP6 downscaling with TSMP) available via GitHub. The configuration maintenance and experiment handling will be documented in the new TSMP documentation on GitHub. With the aforementioned concept, also the static fields and forcing data are not an issue any more and can be maintained even better.
   Hence I am challenging the concept of the predefined setups in a first place, that are shipping with the TSMP interface. The new eTSMP / TSMP2 does not include this anymore either. Perhaps sunset those that do not work at this point in time. Maintain those, which still work (should be the ones from the HPSC TerrSys Fall School). Get rid of the built in setups in the mid term. Provide standard experiments for testing, development, familiarisation, Fall School, through their setups and configs and workflows separately.

   👍React with 👍1Reacted by Daniel Caviedes-Voullième
7. DCaviedesV commented on Mar 7, 2023

   

   DCaviedesV

   on Mar 7, 2023

   ContributorAuthor

   More actions

   @kgoergen, we are clearly aligned. These supported setups shipping with TSMP should be a minimal set of well-known tests.
   I propose these should reduce to:

   • EUR11 CORDEX
   • NRW
   • idealRTD
   • idealScal
   • possibly some variations of the idealised cases (currently in the pipeline)
   • possibly ICON-enabled versions of the above.

   any other suggestions (to all in this conversation)? @jjokella also a case or two supporting PDAF, right?
   Moreover, for those these set of supported cases, we should have a reference solution, as discussed in #168.
8. niklaswr commented on Mar 8, 2023

   

   niklaswr

   on Mar 8, 2023

   Member

   More actions

   I totally agree in reducing the test cases to what they are: tests to check that the installation went well.
   So fine with removing deprecated setups.

   Further, I would suggest reducing the the proposed lists of test-cases even more, and use ideal test-cases only.
   Reasons are:

   i) As @chartick mentioned in #168, the 'real' test-cases does have the problem of being outdated, once the static files does see any update. Even if the static files does (should) not change much, they will most likely change from time to time. And then you have to update the test-cases, which means more maintenance, which may be forgotten, or there simply is no time. With ideal test-cases you don't have this problem.

   ii) Using 'real' test-cases could lead to some confusion. Test-cases should be test cases, an not the starting point for developing new experiments or as source for the static files. We have seen in the past, that people use a test-case as the basis for their experiments. I think it might help if we provide only ideal test-cases, and make it clear that these are just tests.

   I totally agree in reducing the test cases to what they are: tests to check if installation went fine.
   So fine with removing deprecated setups.
9. kgoergen commented on Mar 8, 2023

   

   kgoergen

   on Mar 8, 2023

   Contributor

   More actions

   So for the moment the system as is (including some test experiment setups+configurations as part of the TSMP interface) will remain for for some time, albeit with reduced number of experiments. I see the argument for mainly going to ideal cases. But I am in favour to also provide a test experiment setup/configuration for real data cases, such as a short EUR-11 CORDEX test case. TSMP is and will be used very much also for climate mode continental scale simulations; to get external users started, who most likely do not have access to the experiment management system with current setups and configurations, it is nice to have a starting point. On the mid-term when these experiments are also on github, then we may get rid of these built-in examples. On the long-run, the built-in experiments will vanish anyway. To reduce the maintenance and make the system slimmer and more consistent at this point in time, I would opt that some of the built-in setups/configurations remain, but they not necessarily have to be maintained and be fully up to date with the current reference implementations. Also, we may not need to provide the reference results (i.e., result of running a "test experiment" in debug mode), further reducing necessary maintenance efforts.
10. DCaviedesV commented on Mar 8, 2023

    

    DCaviedesV

    on Mar 8, 2023

    ContributorAuthor

    More actions

    I too see the idealised test cases as the really relevant ones to ship with TSMP, but having at least one (or a couple) of real test cases is important for someone who might want to go into the adventure of setting up a new case. I like the idea of progressively phasing these out, once the maintained setups are publicly available. We can progressively do this.
    I think at this time we do a first clean-up of really deprecated stuff, and put some clear order in these things.

    I do think it is relevant to have the realistic cases that ship with TSMP rather up-to-date with the maintained (production) steups, at least the static fields, to avoid confusion for people who may start using the included test case (e.g. EUR 11) but then change to production setups.

    In terms of the reference results, I would argue it is in fact key to have them, so that we can, eventually deploy automatic benchmarking upon, for example, reviewing a pull request.
11. s-poll commented on Mar 9, 2023

    

    s-poll

    on Mar 9, 2023

    Member

    More actions

    I agree that the number of test cases can/should be reduced. I would also keep one / a few real test cases, as some features are not covered by the existing ideal test cases (e.g. multiple PFTs usage, tile approach) and would not pop up in automatic benchmarking.

    Re cases:

    • The idealLES case uses the LES scheme of COSMO (Herzog-scheme). This test case was using COSMO only and namelist should be fine, but I do not really see a benefit from this case. From my side this could be removed.
    • The multi-scale test case is some sensitivity test case from Prabhakar, where a wrapper is build around the setup_tsmp.ksh. I have never used this case, but looks rather similar to the idealscal.
      .
    • ICON-cases in general: I would favor to have one ICON test case ready. Thus, everyone would have the possibility to test if a PR will destroy TSMP functionality with ICON-CLM3.5-ParFlow. In fact I am not sure if any of the ICON test case are currently running, as they were originally based on env-files, which I can not see in the master anymore. As well, there were some problems with the implemented git-log in the past. I can not recall if these problems are solved. I think until the test experiments setups are fully staged, it would be good to have one ideal and one real test case running.
    • Germany: This test case basically covers the domain and configuration of ICON-D2 for v2.6.2.2. Even though if this the test-case which is documented best, I would sunset it. This test case never run in a coupled mode.
    • icon-ccs: This is an idealized test case using a similar grid between ICON and CLM/PFL. Atmospheric initialization is after Dipankar 2014 with homogeneous land cover. I would like to keep it. I have to take a look into the rmp-file generation for this case, but in general the rmp-file generation works. Alternatively we could put the rmp-files to the datapub.
    • nrw-icon: As the name already suggest it covers the same modeling domain as the nrw case, but using ICON instead of COSMO. This is a real case using ICON in NWP mode. I have to take a look into it, why it is "not working". I would like to keep the case as potential conflicts will most likely pop up in this case.
12. chartick commented on Mar 9, 2023

    

    chartick

    on Mar 9, 2023

    Contributor

    More actions

    Thanks for the clarification. I think both icon-ccs and nrw-icon do not work with Intel because you cannot load nco and cdo with it. I did not try it with GNU. Probably it would be best just to supply the remap files.
13. jjokella commented on Mar 22, 2023

    

    jjokella

    on Mar 22, 2023

    Contributor

    More actions

    ? @jjokella also a case or two supporting PDAF, right?

    In the current status of test cases, the FallSchoolCase should be the standard case using TSMP-PDAF. It does not use however the environment of the blsdsva/setups/ directory, therefore I would leave it for now as a separate HowTo.

    Discussion points for future TSMP-PDAF setups:

    • The FallSchoolCase could be incorporated in the bldsva/setups framework
    • new PDAF testcases could be generated from the standard setups tested during work on this issue
14. DCaviedesV commented on Mar 29, 2023

    

    DCaviedesV

    on Mar 29, 2023

    ContributorAuthor

    More actions

    @jjokella can we open an issue for the TSMP PDAF cases that might be convenient to ship with TSMP? I think otherwise this issue can be closed. Right @chartick ?
15. chartick commented on Mar 30, 2023

    

    chartick

    on Mar 30, 2023

    Contributor

    More actions

    Yes, I think so.
16. chartick

    closed this as completedon Mar 30, 2023
17. jjokella commented on Apr 3, 2023

    

    jjokella

    on Apr 3, 2023

    Contributor

    More actions

    @jjokella can we open an issue for the TSMP PDAF cases that might be convenient to ship with TSMP? I think otherwise this issue can be closed. Right @chartick ?

    I opened #177