Skip to content

New buffer access in xxx (and ffv) functions - remove ipagV from C++ api - #313

Merged
valassi merged 43 commits into
madgraph5:masterfrom
valassi:klas3bis
Dec 13, 2021
Merged

valassi merged 43 commits into
madgraph5:masterfrom
valassi:klas3bis

Conversation

@valassi

@valassi valassi commented Dec 11, 2021

Copy link
Copy Markdown
Member

This is a WIP PR for the ideas discussed in the first bullet of #175 (comment)

The idea is to start separating CUDA and C++, so that CUDA does a data access in a large buffer inside the XX functions (and eventually in the FFV functions to split into smaller kernels), while C++ takes a single event or at most a SIMD vector of events.

In practice this means removing ipagV from many interfaces.

Note that ipar is still there for CUDA - this cannot be changed unless/until a new memory structure is introduced, as discussed in the SOAOSOA #309

Note also that ip4 is still in the interface of the bufferAccessMomenta. I think this is unavoidable for CUDA: we want to have an AOSOA for events and 4-components, with a striding not fixed a priori. This also does not harm for c++: it could be avoided as one could just get an fptype_sv[4] and index it in a very intuitive way, but it does not harm to have it as a function.

The same idea could be used eventually for splitting the kernels (#310): it would be enough to add for instance bufferAccessWavefunction( fptype* buffer, int iw6 ) wherethe iw6 index is passed in the same way as the ip4 now.

This is WIP, a lot to do, but the basic idea is there now.

…anges

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.940136e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113848e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.734095 sec
       393,467,151      cycles:u                  #    0.408 GHz
       709,346,008      instructions:u            #    1.80  insn per cycle
       1.027992796 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.303097e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.221896 sec
    19,053,500,298      cycles:u                  #    2.633 GHz
    48,696,237,229      instructions:u            #    2.56  insn per cycle
       7.240189441 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.908897e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.635227 sec
     8,968,023,902      cycles:u                  #    2.457 GHz
    16,446,697,673      instructions:u            #    1.83  insn per cycle
       3.652880396 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
…892107

None of these is still used anywhere - performance remains the same
…1/eemumu - all ok, good performances

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.927727e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.112874e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.934527 sec
       759,358,027      cycles:u                  #    0.664 GHz
     1,460,203,275      instructions:u            #    1.92  insn per cycle
       1.234834519 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.312128e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.242481 sec
    19,031,081,712      cycles:u                  #    2.622 GHz
    48,696,253,110      instructions:u            #    2.56  insn per cycle
       7.266153195 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.916675e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.672978 sec
     9,027,104,402      cycles:u                  #    2.445 GHz
    16,446,713,690      instructions:u            #    1.82  insn per cycle
       3.695645301 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
…n pointers

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.969003e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.111533e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.959843 sec
       758,028,751      cycles:u                  #    0.657 GHz
     1,460,327,449      instructions:u            #    1.93  insn per cycle
       1.273118826 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.311472e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.291478 sec
    19,185,553,275      cycles:u                  #    2.625 GHz
    48,696,253,284      instructions:u            #    2.54  insn per cycle
       7.314469200 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.911322e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.675334 sec
     9,028,582,206      cycles:u                  #    2.444 GHz
    16,446,713,774      instructions:u            #    1.82  insn per cycle
       3.697490131 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
@valassi valassi changed the title New buffer access in xxx (and ffv) functions New buffer access in xxx (and ffv) functions - remove ipagV from C++ api Dec 11, 2021
@valassi valassi self-assigned this Dec 11, 2021
@valassi
valassi marked this pull request as draft December 11, 2021 10:25
@valassi valassi changed the title New buffer access in xxx (and ffv) functions - remove ipagV from C++ api WIP: New buffer access in xxx (and ffv) functions - remove ipagV from C++ api Dec 11, 2021
…k to fptype_sv in xxx functions

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.967032e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.114019e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     1.096448 sec
       773,971,897      cycles:u                  #    0.663 GHz
     1,458,037,221      instructions:u            #    1.88  insn per cycle
       1.398319897 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.310183e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.262135 sec
    19,070,073,943      cycles:u                  #    2.619 GHz
    48,696,258,394      instructions:u            #    2.55  insn per cycle
       7.286654464 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.916607e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.691983 sec
     9,051,495,450      cycles:u                  #    2.438 GHz
    16,446,719,020      instructions:u            #    1.82  insn per cycle
       3.716089213 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
…- add implicit conversion p4type_sv to fptype_sv*

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.953641e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.114671e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.935976 sec
       768,960,916      cycles:u                  #    0.661 GHz
     1,467,936,740      instructions:u            #    1.91  insn per cycle
       1.248062395 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.311101e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.254034 sec
    19,066,177,157      cycles:u                  #    2.622 GHz
    48,696,258,582      instructions:u            #    2.55  insn per cycle
       7.278082466 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.905104e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.687230 sec
     9,047,700,943      cycles:u                  #    2.440 GHz
    16,446,718,506      instructions:u            #    1.82  insn per cycle
       3.711097818 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
…sMomenta - used for cuda, not yet fully in c++

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.958293e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.114393e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.766816 sec
       414,707,867      cycles:u                  #    0.412 GHz
       722,267,590      instructions:u            #    1.74  insn per cycle
       1.066649196 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.305823e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.283923 sec
    19,045,988,578      cycles:u                  #    2.609 GHz
    48,696,259,263      instructions:u            #    2.56  insn per cycle
       7.308598173 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.925262e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.674796 sec
     9,022,915,294      cycles:u                  #    2.442 GHz
    16,446,719,005      instructions:u            #    1.82  insn per cycle
       3.698812056 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 6.057387e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.141808e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     1.010526 sec
       765,676,850      cycles:u                  #    0.666 GHz
     1,459,667,841      instructions:u            #    1.91  insn per cycle
       1.311940329 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.311844e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.251677 sec
    19,048,456,703      cycles:u                  #    2.621 GHz
    48,696,258,480      instructions:u            #    2.56  insn per cycle
       7.275597617 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.904536e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.687137 sec
     9,051,969,019      cycles:u                  #    2.442 GHz
    16,446,719,045      instructions:u            #    1.82  insn per cycle
       3.710988915 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.911154e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.105379e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.765036 sec
       418,055,605      cycles:u                  #    0.416 GHz
       716,983,446      instructions:u            #    1.72  insn per cycle
       1.065857170 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.312343e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.235976 sec
    19,044,928,715      cycles:u                  #    2.624 GHz
    48,696,258,019      instructions:u            #    2.56  insn per cycle
       7.260246122 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  636) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.929951e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.709707 sec
     9,109,382,472      cycles:u                  #    2.442 GHz
    16,446,719,092      instructions:u            #    1.81  insn per cycle
       3.733559283 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2704) (512y:   52) (512z:    0)
=========================================================================
…ok (~2% slower in c++?)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.924484e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113887e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.762733 sec
       413,030,276      cycles:u                  #    0.412 GHz
       721,969,023      instructions:u            #    1.75  insn per cycle
       1.062606040 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.300178e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.289449 sec
    19,175,817,992      cycles:u                  #    2.624 GHz
    48,343,933,614      instructions:u            #    2.52  insn per cycle
       7.313170955 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  574) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.820191e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.741431 sec
     9,175,766,443      cycles:u                  #    2.441 GHz
    16,248,536,622      instructions:u            #    1.77  insn per cycle
       3.765174905 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2754) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.963277e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.114171e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.965056 sec
       764,711,277      cycles:u                  #    0.664 GHz
     1,461,264,678      instructions:u            #    1.91  insn per cycle
       1.265600027 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.292914e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.312846 sec
    19,242,259,949      cycles:u                  #    2.625 GHz
    48,343,933,778      instructions:u            #    2.51  insn per cycle
       7.336570575 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  574) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.844702e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.719313 sec
     9,132,329,454      cycles:u                  #    2.442 GHz
    16,248,536,673      instructions:u            #    1.78  insn per cycle
       3.742908010 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2754) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.975366e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.115482e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.941425 sec
       762,940,296      cycles:u                  #    0.663 GHz
     1,468,151,082      instructions:u            #    1.92  insn per cycle
       1.242135101 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.300589e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.280185 sec
    19,148,003,713      cycles:u                  #    2.623 GHz
    48,318,767,984      instructions:u            #    2.52  insn per cycle
       7.304646286 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  574) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.832714e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.704604 sec
     9,094,062,506      cycles:u                  #    2.441 GHz
    16,248,536,525      instructions:u            #    1.79  insn per cycle
       3.728801864 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2754) (512y:   52) (512z:    0)
=========================================================================
…UNCTIONS...

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.996332e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.120328e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.975801 sec
       761,377,731      cycles:u                  #    0.656 GHz
     1,466,316,940      instructions:u            #    1.93  insn per cycle
       1.279572623 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.297201e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.301089 sec
    19,198,479,392      cycles:u                  #    2.623 GHz
    48,318,768,371      instructions:u            #    2.52  insn per cycle
       7.325079977 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  574) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.826184e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.710541 sec
     9,097,646,008      cycles:u                  #    2.439 GHz
    16,248,537,127      instructions:u            #    1.79  insn per cycle
       3.733582744 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2754) (512y:   52) (512z:    0)
=========================================================================
…ower

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.954583e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113358e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.761552 sec
       413,546,907      cycles:u                  #    0.414 GHz
       727,127,575      instructions:u            #    1.76  insn per cycle
       1.061515791 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.293610e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.326218 sec
    19,281,456,331      cycles:u                  #    2.625 GHz
    48,318,767,477      instructions:u            #    2.51  insn per cycle
       7.349502477 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.695919e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.748591 sec
     9,197,162,277      cycles:u                  #    2.441 GHz
    16,688,941,210      instructions:u            #    1.81  insn per cycle
       3.772120405 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2752) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.918101e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.111580e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.769473 sec
       411,091,298      cycles:u                  #    0.410 GHz
       717,754,322      instructions:u            #    1.75  insn per cycle
       1.069596435 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.301696e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.285865 sec
    19,178,043,588      cycles:u                  #    2.625 GHz
    48,318,767,095      instructions:u            #    2.52  insn per cycle
       7.309316569 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.512116e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.718906 sec
     9,080,082,207      cycles:u                  #    2.429 GHz
    16,386,951,251      instructions:u            #    1.80  insn per cycle
       3.743047818 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2794) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.969161e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113501e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.773621 sec
       415,155,217      cycles:u                  #    0.411 GHz
       717,379,589      instructions:u            #    1.73  insn per cycle
       1.074332216 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.301150e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.281312 sec
    19,151,429,304      cycles:u                  #    2.623 GHz
    48,318,768,412      instructions:u            #    2.52  insn per cycle
       7.304786269 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.818740e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.703873 sec
     9,089,320,140      cycles:u                  #    2.441 GHz
    16,248,536,398      instructions:u            #    1.79  insn per cycle
       3.727230595 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2754) (512y:   52) (512z:    0)
=========================================================================
…lty if anything (now 4.86?!)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.936738e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113886e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.765109 sec
       413,197,548      cycles:u                  #    0.412 GHz
       723,218,856      instructions:u            #    1.75  insn per cycle
       1.064332586 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.302016e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.307847 sec
    19,231,018,339      cycles:u                  #    2.625 GHz
    48,318,767,089      instructions:u            #    2.51  insn per cycle
       7.331490647 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.855278e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.715297 sec
     9,125,943,724      cycles:u                  #    2.443 GHz
    16,273,702,614      instructions:u            #    1.78  insn per cycle
       3.738512895 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.930947e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.110873e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.758637 sec
       411,453,190      cycles:u                  #    0.413 GHz
       717,627,965      instructions:u            #    1.74  insn per cycle
       1.058214819 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.301208e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.281200 sec
    19,166,963,587      cycles:u                  #    2.625 GHz
    48,318,767,492      instructions:u            #    2.52  insn per cycle
       7.304740930 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
WARNING! arbitrary array
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.299171e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.870550 sec
     9,473,318,297      cycles:u                  #    2.434 GHz
    17,506,839,400      instructions:u            #    1.85  insn per cycle
       3.895081499 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2783) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.935073e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113731e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.761975 sec
       411,482,675      cycles:u                  #    0.410 GHz
       718,980,829      instructions:u            #    1.75  insn per cycle
       1.064540223 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.301737e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.274357 sec
    19,150,038,986      cycles:u                  #    2.625 GHz
    48,318,767,059      instructions:u            #    2.52  insn per cycle
       7.298094984 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
WARNING! arbitrary array
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.491809e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.718218 sec
     9,068,572,731      cycles:u                  #    2.429 GHz
    16,500,197,400      instructions:u            #    1.82  insn per cycle
       3.741645759 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2784) (512y:   52) (512z:    0)
=========================================================================
…ent check every time)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.958651e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.111818e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.768364 sec
       414,840,251      cycles:u                  #    0.413 GHz
       716,621,300      instructions:u            #    1.73  insn per cycle
       1.068316030 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.292061e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.321030 sec
    19,252,455,161      cycles:u                  #    2.624 GHz
    48,318,767,801      instructions:u            #    2.51  insn per cycle
       7.344887374 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.843689e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.702359 sec
     9,079,231,621      cycles:u                  #    2.440 GHz
    16,273,702,715      instructions:u            #    1.79  insn per cycle
       3.726554931 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
…rate MemoryAccess.h - all ok

This will need a much more throught cleanup, once we agree better what is needed in the Fortran bridge

But for the moment this is a reasonable separation of data access and computations

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.525339e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.162881e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     1.123886 sec
       785,576,398      cycles:u                  #    0.880 GHz
     1,490,945,195      instructions:u            #    1.90  insn per cycle
       1.427680604 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.301289e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.290284 sec
    19,162,611,591      cycles:u                  #    2.621 GHz
    48,318,763,032      instructions:u            #    2.52  insn per cycle
       7.316508951 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.851138e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.716292 sec
     9,109,552,318      cycles:u                  #    2.436 GHz
    16,273,698,120      instructions:u            #    1.79  insn per cycle
       3.743027573 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
…sAmplitude (trivial, for the moment) - all ok

(No functional change for the moment - kernelAccessAmplitude is still the same for cuda and c++)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.578246e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.176616e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     1.033650 sec
       780,589,876      cycles:u                  #    0.661 GHz
     1,496,096,610      instructions:u            #    1.92  insn per cycle
       1.336930631 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.302509e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.292469 sec
    19,173,073,325      cycles:u                  #    2.621 GHz
    48,318,763,508      instructions:u            #    2.52  insn per cycle
       7.319408337 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.847707e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.713602 sec
     9,102,914,462      cycles:u                  #    2.437 GHz
    16,273,698,804      instructions:u            #    1.79  insn per cycle
       3.739403482 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
…RD_PATH in epoch1 as done in epochX (issue madgraph5#194)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.430542e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.112191e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.790035 sec
       432,265,595      cycles:u                  #    0.419 GHz
       752,034,458      instructions:u            #    1.74  insn per cycle
       1.093489337 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.303264e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.297318 sec
    19,169,051,850      cycles:u                  #    2.618 GHz
    48,318,765,409      instructions:u            #    2.52  insn per cycle
       7.324402289 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.805005e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.741404 sec
     9,120,132,021      cycles:u                  #    2.422 GHz
    16,273,702,231      instructions:u            #    1.78  insn per cycle
       3.769439753 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.431251e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.111897e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     1.132435 sec
     1,061,482,814      cycles:u                  #    0.809 GHz
     2,042,247,419      instructions:u            #    1.92  insn per cycle
       1.449279763 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.303611e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.302474 sec
    19,182,786,382      cycles:u                  #    2.618 GHz
    48,318,767,121      instructions:u            #    2.52  insn per cycle
       7.331050665 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.839633e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.732373 sec
     9,131,519,732      cycles:u                  #    2.432 GHz
    16,273,702,812      instructions:u            #    1.78  insn per cycle
       3.760368811 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
…r the ifdef MGONGPU_HARDCODE_CIPC)

On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
=========================================================================
Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
EvtsPerSec[MatrixElems] (3) = ( 5.450328e+08                 )  sec^-1
EvtsPerSec[MECalcOnly] (3a) = ( 1.113992e+09                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     0.791362 sec
       435,349,317      cycles:u                  #    0.420 GHz
       752,035,433      instructions:u            #    1.73  insn per cycle
       1.098707209 seconds time elapsed
==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
=========================================================================
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 1.302442e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     7.304491 sec
    19,181,226,930      cycles:u                  #    2.618 GHz
    48,318,766,149      instructions:u            #    2.52  insn per cycle
       7.331533054 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
-------------------------------------------------------------------------
Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
OMP threads / `nproc --all` = 1 / 4
EvtsPerSec[MECalcOnly] (3a) = ( 4.855146e+06                 )  sec^-1
MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
TOTAL       :     3.729151 sec
     9,132,014,260      cycles:u                  #    2.434 GHz
    16,273,703,332      instructions:u            #    1.78  insn per cycle
       3.756437859 seconds time elapsed
=Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
=========================================================================
… ipagV, use the kernelAccess functions) - all ok
…ier on in HelAmps_sm.h and add Amplitude function
…ions earlier on in HelAmps_sm.h and add Amplitude

    On itscrd70.cern.ch [CPU: Intel(R) Xeon(R) Silver 4216 CPU] [GPU: 1x Tesla V100S-PCIE-32GB]:
    =========================================================================
    Process                     = EPOCH1_EEMUMU_CUDA [nvcc 11.1.105 (gcc 10.2.0)] [inlineHel=0] [hardcodeCIPC=0]
    FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
    EvtsPerSec[MatrixElems] (3) = ( 5.441266e+08                 )  sec^-1
    EvtsPerSec[MECalcOnly] (3a) = ( 1.120748e+09                 )  sec^-1
    MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
    TOTAL       :     0.987913 sec
           792,480,290      cycles:u                  #    0.670 GHz
         1,496,770,892      instructions:u            #    1.89  insn per cycle
           1.292641075 seconds time elapsed
    ==PROF== Profiling "sigmaKin": launch__registers_per_thread 130
    ==PROF== Profiling "sigmaKin": sm__sass_average_branch_targets_threads_uniform.pct 100%
    =========================================================================
    Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
    FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
    Internal loops fptype_sv    = SCALAR ('none': ~vector[1], no SIMD)
    OMP threads / `nproc --all` = 1 / 4
    EvtsPerSec[MECalcOnly] (3a) = ( 1.300619e+06                 )  sec^-1
    MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
    TOTAL       :     7.324798 sec
        19,215,567,029      cycles:u                  #    2.616 GHz
        48,318,766,461      instructions:u            #    2.51  insn per cycle
           7.353073181 seconds time elapsed
    =Symbols in CPPProcess.o= (~sse4:  576) (avx2:    0) (512y:    0) (512z:    0)
    -------------------------------------------------------------------------
    Process                     = EPOCH1_EEMUMU_CPP [gcc 10.2.0] [inlineHel=0] [hardcodeCIPC=0]
    FP precision                = DOUBLE (NaN/abnormal=0, zero=0)
    Internal loops fptype_sv    = VECTOR[4] ('512y': AVX512, 256bit) [cxtype_ref=YES]
    OMP threads / `nproc --all` = 1 / 4
    EvtsPerSec[MECalcOnly] (3a) = ( 4.850439e+06                 )  sec^-1
    MeanMatrixElemValue         = ( 1.371706e-02 +- 3.270315e-06 )  GeV^0
    TOTAL       :     3.726553 sec
         9,120,667,109      cycles:u                  #    2.432 GHz
        16,273,702,551      instructions:u            #    1.78  insn per cycle
           3.754039059 seconds time elapsed
    =Symbols in CPPProcess.o= (~sse4:    0) (avx2: 2762) (512y:   52) (512z:    0)
    =========================================================================
…mu epochX - most likely it will not be needed
…est manual - all ok, no change in performance
@valassi
valassi marked this pull request as ready for review December 13, 2021 13:09
@valassi valassi changed the title WIP: New buffer access in xxx (and ffv) functions - remove ipagV from C++ api New buffer access in xxx (and ffv) functions - remove ipagV from C++ api Dec 13, 2021
@valassi

valassi commented Dec 13, 2021

Copy link
Copy Markdown
Member Author

This is now complete thus far and can be merged. Main changes

  • removed ipagV and ipar from c++ internal APIs
  • started differentiating cuda and c++ (cuda xxx functions dela with large arrays, c++ deals with a single event or event vector)
  • renamed and moved indexing functions to MemoryAccess.h
  • for the moment, I did keep the complex logic to deal with aligned, unaligned and arbitrary arrays (it can be simplified later on), I actually removed sanity checks and assumptions, and reenabled alignment checks at every call, the overhead is minimal

@valassi

valassi commented Dec 13, 2021

Copy link
Copy Markdown
Member Author

Self merging

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant