Treffer: An improved deep reinforcement learning method for flink configuration parameter tuning.

Apache Flink is a widely used big data processing system integrating batch and streaming. For batch processing, the setting of Flink configuration parameters is an important factor affecting the performance of Flink workloads. However, due to the complex interaction between configuration parameters, manually tuning the parameters to achieve superior performance is tedious. In this paper, we propose an improved deep reinforcement learning method for Flink configuration parameter tuning. First, we train a performance prediction model based on random forest to reliably predict the performance corresponding to Flink's specific configuration. Then, we improve Deep Q-Networks (DQN) algorithm so that the search agent interacts iteratively with the performance prediction model to optimize Flink's key performance configuration parameters in an automated manner. Experimental results show that our configuration parameter optimization process converges, and the optimized configuration obtained achieves better performance than the default configuration. For three typical Flink batch processing workloads, compared with the default configuration, the performance corresponding to the optimized configuration is improved by an average of 36.96, 23.44, and 23.75% in different scales of data sets, respectively. And the overall performance of the three workloads is improved by an average of 28.05%, which indicates the effectiveness of our method. [ABSTRACT FROM AUTHOR]