clickhouse merge挂住场景之一

clickhouse merge挂住场景之一

在一次测试过程中，单节点建了一张mergetree表，8C32G环境，并发80线程小文件写入clickhouse，出现了一次merge挂住，包括系统表在内的所有表都无法进行merge。但是从后台的trace日志和gdb来看已经执行了selectPartsToMerge函数，但是最终没有调度到mergeSelectedParts。trace日志的表现如下：

trace日志一直打印：(MergerMutator): Selected 9 parts from 202201_575_575_0 to 202201_583_583_0。但是没有进一步的执行merge操作。

反复观察BackgroundPoolTask的统计值，发现该值一直是16，大小相当于background_pool_size，

tesspmrB0001.mrs-dviw.com :) select metric , value from system.metrics where metric like '%Pool%';

SELECT
    metric,
    value
FROM system.metrics
WHERE metric LIKE '%Pool%'

Query id: 359e9bfe-fe4a-4590-908f-bd6a1b90510b

┌─metric──────────────────────────────────┬─value─┐
│ BackgroundPoolTask                      │    16 │
│ BackgroundFetchesPoolTask               │     0 │
│ BackgroundMovePoolTask                  │     0 │
│ BackgroundSchedulePoolTask              │     1 │
│ BackgroundBufferFlushSchedulePoolTask   │     0 │
│ BackgroundDistributedSchedulePoolTask   │     0 │
│ BackgroundMessageBrokerSchedulePoolTask │     0 │
└─────────────────────────────────────────┴───────┘

从selectPartsToMerge往回追溯线程池的调用流程：

IBackgroundJobExecutor::jobExecutingTask
  BackgroundJobsExecutor::getBackgroundJob
    StorageMergeTree::getDataProcessingJob
      StorageMergeTree::selectPartsToMerge
  StorageMergeTree::mergeSelectedParts  // 在IBackgroundJobExecutor::jobExecutingTask中通过job调用

截取IBackgroundJobExecutor::jobExecutingTask部分调用逻辑可以看到：

void IBackgroundJobExecutor::jobExecutingTask()
try
{
    auto job_and_pool = getBackgroundJob();
    if (job_and_pool) /// If we have job, then try to assign into background pool
    {
        auto & pool_config = pools_configs[job_and_pool->pool_type];
        /// If corresponding pool is not full increment metric and assign new job
        if (incrementMetricIfLessThanMax(CurrentMetrics::values[pool_config.tasks_metric], pool_config.max_pool_size))
        {
            try /// this try required because we have to manually decrement metric
            {
                pools[job_and_pool->pool_type].scheduleOrThrowOnError([this, pool_config, job{std::move(job_and_pool->job)}] ()
                {
                    try /// We don't want exceptions in background pool
                    {
                        bool job_success = job();
                        /// Job done, decrement metric and reset no_work counter
                        CurrentMetrics::values[pool_config.tasks_metric]--;

                        if (job_success)
                        {
                            /// Job done, new empty space in pool, schedule background task
                            runTaskWithoutDelay();
                        }
                        else
                        {
                            /// Job done, but failed, schedule with backoff
                            scheduleTask(/* with_backoff = */ true);
                        }

                    }

incrementMetricIfLessThanMax对比BackgroundPoolTask和background_pool_size的大小，在BackgroundPoolTask小于background_pool_size的情况下才会进一步调用 job()，也就是mergeSelectedParts，这也是为什么我们在日志中只能看到selectPartsToMerge部分的日志。但是具体为什么BackgroundPoolTask持续撑满，目前还没找到原因，当时将测试环境重启后环境恢复，merge也恢复正常。