MAX_NUM_RUNTIME_FILTERS Query Option (CDH 5.7 or higher only)
The MAX_NUM_RUNTIME_FILTERS query option sets an upper limit on the number of runtime filters that can be produced for each query.
Added in: CDH 5.7.0 (Impala 2.5.0)
Each runtime filter imposes some memory overhead on the query. Depending on the setting of the RUNTIME_BLOOM_FILTER_SIZE query option, each filter might consume between 1 and 16 megabytes per plan fragment. There are typically 5 or fewer filters per plan fragment.
Impala evaluates the effectiveness of each filter, and keeps the ones that eliminate the largest number of partitions or rows. Therefore, this setting can protect against potential problems due to excessive memory overhead for filter production, while still allowing a high level of optimization for suitable queries.
Because the runtime filtering feature is enabled by default only for local processing, the other filtering-related query options have the greatest effect when used in combination with the setting RUNTIME_FILTER_MODE=GLOBAL.
Because the runtime filtering feature applies mainly to resource-intensive and long-running queries, only adjust this query option when tuning long-running queries involving some combination of large partitioned tables and joins involving large tables.