Very little GPU utilization on Tesla P100

[karlkashofer]

I compiled with GPUARCH=sm_60 which should be the right arch for Tesla P100.
Feeding reads (human genomic reads, avg length 5kb) to mm2-gp works and performs mapping, but the GPU is only used for very short periods of time (seconds) while one CPU thread is working at 100% for a very long time (~5min) for each batch of ~35k reads.

[M::main::7.899*1.00]` loaded/built the index for 218 target sequence(s)
[M::mm_mapopt_update::9.257*1.00] mid_occ = 694
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 218
[M::mm_idx_stat::10.171*1.00] distinct minimizers: 100168976 (38.82% are singletons); average occurrences: 5.519; average spacing: 5.610; total length: 3100885153
[Info] max anchors per stream: 336880000, max range grid 347161 max_num_cut 994849 long_seg_buffer_size 417376000
[Info] Host Malloc Pinned Memory Size 5.97 GB
[Info] Host Malloc Pinned Memory Size 2.33 GB (long seg)
[Info] GPU free mem: 8.380981 GB, total mem: 15.894775 GB (before alloc long seg buffer) 
[Info] GPU free mem: 0.984497 GB, total mem: 15.894775 GB
[Info::init_stream_gpu] gpu initialized for chaining with config mi210_below50k_config.json
[Info::init_stream_gpu] Compile time config: 
                 USE CUDA
                 MAX MICRO BATCH        8
[Info] plchain_cal_score_async (gpu/plchain.cu:322) Launching Batch: n_read 35133, total anchors 336778391 (mem usage: 99.97%)
----------------------------------------------------------------------------
                  Short0     Long
----------------------------------------------------------------------------
Mem Usage  =      99.97%      4.75 %
Runtime(s) =        0.55        0.58
BW (Ma/s)  =      578.58       34.23
BW(Mpair/s)=     9065.30    15180.81
----------------------------------------------------------------------------
[Info] plchain_cal_score_async (gpu/plchain.cu:322) Launching Batch: n_read 31414, total anchors 336876054 (mem usage: 100.00%)

In absence of a conf file i cobbled this together which makes mm2-gp run, albeit very inefficiently:

{
    "//config is for": "aac cloud. Fits one batch + 5% x 4 long buffer avg_read_n 10k",
    "num_streams": 1,
    "min_n": 1, 
    "//min_n": "queries with less anchors will be handled on cpu",
    "long_seg_buffer_size": 417376000,
    "max_total_n": 336880000, 
    "max_read": 336880,
    "avg_read_n": 50000,
    "//avg_read_n": "expect average number of anchors per read, not used if max_total_n and max_read are specified",
    "range_kernel": {
        "blockdim": 512,
        "cut_check_anchors": 10,
        "//cut_check_anchors": "Number of anchors to check to attemp a cut",
        "anchor_per_block": 32768,
        "//anchor_per_block": "Number of anchors each block handle. Must be int * blockdim"
    },
    "score_kernel": {
        "micro_batch": 1,
        "mid_blockdim": 512,
        "//static options for mid_blockdim": "128/256/512/1024",
        "short_griddim": 3328,
        "mid_griddim": 3328,
        "long_griddim": 208,
        "//normal reads benefit from more blocks": "long_griddim = 2 * num of CUs",
        "long_seg_cutoff": 20,
        "mid_seg_cutoff": 3
    }
}

Would you have any idea if that can be fixed with the gpu config file ?
Any recommendations on what to do ?