Title | Metric Name | Metric Unit | Metric Meaning |
number of Shuffle Servers | total_server_num | Count | Number of Shuffle Servers registered to the Coordinator |
| exclude_server_num | Count | Number of ShuffleServers in the exclude file list |
| unhealthy_server_num | Count | Number of unhealthy Shuffle Servers |
number of applications | running_app_num | Count | Number of running applications registered to the Coordinator |
| total_app_num | Count | Total number of applications |
Request | total_access_request | Count | Total number of requests |
| total_candidates_denied_request | Count | Number of requests denied due to accessId not being in the candidate list |
| total_load_denied_request | Count | Number of requests denied due to excessive memory load on the ShuffleServer |
| total_quota_denied_request | Count | Number of requests denied due to user-level Application quota quantity control |
| total_banned_denied_request | Count | Number of requests denied due to being included in the banned Id list |
gRPC thread pool | active_threads | Count | Number of active threads in the grpc thread pool |
| blocking_queue_size | Count | Number of queued tasks in the grpc thread pool queue |
gRPC connections | grpc_server_connection_number | Count | Current number of gRPC connections |
number of gRPC requests | grpc_open | Count | Number of open gRPC requests |
| grpc_total | Count | Total gRPC number of requests |
| grpc_heartbeat | Count | Number of in-process heartbeat gRPC requests |
| grpc_get_shuffle_assignments_total | Count | Total number of get_shuffle_assignments gRPC requests |
| grpc_get_shuffle_assignments | Count | Number of in-process get_shuffle_assignments gRPC requests |
| grpc_heartbeat_total | Count | Total number of heartbeat gRPC requests |
| execution_less_than_1s_rpc_count | Count | Number of rpc requests with execution time less than 1 second |
| heartbeat_rpc_count | Count | Total number of rpc requests |
| less_than_1s_heartbeat_rpc_cnt | Count | Number of heartbeat rpc requests with execution time less than 1 second |
CPU utilization | ProcessCpuLoad | % | CPU utilization |
number of GCs | YGC | Count | Young GC count |
| FGC | Count | Full GC count |
GC time | FGCT | s | Full GC consumption time |
| GCT | s | Garbage collection consumption time |
| YGCT | s | Young GC consumption time |
Memory zone proportion | S0 | % | Percentage of used Survivor 0 memory |
| E | % | Percentage of used Eden memory |
| CCS | % | Percentage of used compressed class space memory |
| S1 | % | Percentage of used Survivor 1 memory |
| O | % | Percentage of used Old memory |
| M | % | Percentage of used Metaspace memory |
JVM memory | MemHeapUsedM | MBytes | Used heap memory |
| MemNonHeapUsedM | MBytes | Used non-heap memory |
| MemHeapCommittedM | MBytes | Committed heap memory |
| MemNonHeapCommittedM | MBytes | Committed non-heap memory |
| MemHeapMaxM | MBytes | Maximum heap memory |
| MemHeapInitM | MBytes | Initial heap memory |
| MemNonHeapInitM | MBytes | Initial non-heap memory |
Number of file descriptors | MaxFileDescriptorCount | Count | Maximum number of file descriptors |
| OpenFileDescriptorCount | Count | Number of open file descriptors |
JVM number of threads | PeakThreadCount | Count | Peak number of threads |
| ThreadCount | Count | Total number of threads |
| DaemonThreadCount | Count | Daemon number of threads |
Title | Metric Name | Metric Unit | Metric Meaning |
Current number of applications | app_num_with_node | Count | Number of applications currently storing data on the ShuffleServer |
Buffer | allocated_buffer_size | Bytes | Amount of memory allocated for storing data |
| in_flush_buffer_size | Bytes | Amount of memory pending flush to disk |
| used_buffer_size | Bytes | The amount of memory used to store shuffle data, with its upper limit determined by rss.server.buffer.capacity. |
| read_used_buffer_size | Bytes | The amount of memory used to cache shuffle read data, with its upper limit determined by rss.server.read.buffer.capacity. |
| total_expired_preAllocated_buffer_num | Count | Number of expired preAllocatedBuffers |
| require_buffer_count | Count | Number of preAllocatedBuffer requests |
Data | total_received_data | Bytes | Amount of data received by the ShuffleServer |
| total_write_data | Bytes | Amount of data flushed by the ShuffleServer to disk/remote storage |
| total_delete_data | Bytes | Amount of storage space freed by deleting Application data from disk/HDFS |
Number of data deletions | total_delete_file_num | Count | Total number of deleted files |
Direct memory | used_direct_memory_size | Bytes | Amount of used direct memory |
| used_by_netty | Bytes | Amount of direct memory used by Netty |
| used_by_grpc_netty | Bytes | Amount of direct memory used by grpc_netty |
Event | event_queue_size | Count | Number of events currently in the ShuffleServer flush queue |
| total_failed_written_event_num | Count | Number of events that failed to be persisted |
| total_dropped_event_num | Count | Number of events dropped due to non-existent application or flush timeout |
Flush count | total_flush_file_num | Count | Total number of files flushed by the ShuffleServer to disk/remote storage |
Flush count | flush_file_num | Count | Number of files currently being flushed |
| total_write_num | Count | Number of times the ShuffleServer flushes data to disk/remote storage |
| storage_total_write_local | Count | Total number of writes to local storage |
| storage_retry_write_local | Count | Number of retried writes to local storage |
| storage_failed_write_local | Count | Number of failed writes to local storage |
| storage_success_write_local | Count | Number of successful writes to local storage |
| local_file_event_flush_num | Count | Number of flushes to local storage |
| hadoop_event_flush_num | Count | Number of flushes to Hadoop |
| flush_exceed_high_water_count | Count | Number of flushes triggered by buffer data volume exceeding the high-water mark (the high-water mark is controlled by rss.server.memory.shuffle.highWaterMark.percentage) |
| flush_exceed_buffer_threshold_count | Count | Number of flushes triggered by buffer data volume exceeding the threshold (the threshold is controlled by rss.server.single.buffer.flush.threshold) |
| flush_exceed_block_num_count | Count | Number of flushes triggered by number of buffer blocks exceeding the threshold (the threshold is controlled by rss.server.single.buffer.flush.blocksNumberThreshold) |
Flush data volume | total_hdfs_write_data | Bytes | Amount of data written to HDFS |
Flush thread pool | localfile_flush_thread_pool_queue_size | Count | Number of queued tasks in the local Flush thread pool |
| read_local_data_file_thread_num | Count | Number of current threads reading local storage data |
| read_local_index_file_thread_num | Count | Number of current threads reading local storage index |
| read_memory_data_thread_num | Count | Number of current threads reading memory data |
| localfile_writing_thread_num | Count | Number of threads flushing to local storage |
| hdfs_writing_thread_num | Count | Number of threads flushing to HDFS |
gRPC Process latency | send_shuffle_data_latency_50 | s | 50th percentile latency of the grpc send_shuffle_data process |
| send_shuffle_data_latency_90 | s | 90th percentile latency of the grpc send_shuffle_data process |
| send_shuffle_data_latency_99 | s | 99th percentile latency of the grpc send_shuffle_data process |
| send_shuffle_data_latency_sum | s | Total latency of the grpc send_shuffle_data process |
| get_local_shuffle_data_latency_50 | s | 50th percentile latency of the grpc get_local_shuffle_data process |
| get_local_shuffle_data_latency_90 | s | 90th percentile latency of the grpc get_local_shuffle_data process |
| get_local_shuffle_data_latency_99 | s | 99th percentile latency of the grpc get_local_shuffle_data process |
| get_local_shuffle_data_latency_sum | s | Total latency of the grpc get_local_shuffle_data process |
| get_memory_shuffle_data_latency_50 | s | 50th percentile latency of the gRPC get_memory_shuffle_data process |
| get_memory_shuffle_data_latency_90 | s | 90th percentile latency of the gRPC get_memory_shuffle_data process |
| get_memory_shuffle_data_latency_99 | s | 99th percentile latency of the gRPC get_memory_shuffle_data process |
| get_memory_shuffle_data_latency_sum | s | Total latency of the gRPC get_memory_shuffle_data process |
| get_multipart_shuffle_res_latency_50 | s | 50th percentile latency of the gRPC get_shuffle_result_for_multi_part process |
| get_multipart_shuffle_res_latency_90 | s | 90th percentile latency of the gRPC get_shuffle_result_for_multi_part process |
| get_multipart_shuffle_res_latency_99 | s | 99th percentile latency of the gRPC get_shuffle_result_for_multi_part process |
| get_multipart_shuffle_res_latency_sum | s | Total latency of the gRPC get_shuffle_result_for_multi_part process |
| report_shuffle_result_latency_50 | s | 50th percentile latency of the gRPC report_shuffle_result process |
| report_shuffle_result_latency_90 | s | 90th percentile latency of the gRPC report_shuffle_result process |
| report_shuffle_result_latency_99 | s | 99th percentile latency of the gRPC report_shuffle_result process |
| report_shuffle_result_latency_sum | s | Total latency of the gRPC report_shuffle_result process |
| get_sorted_shuffle_data_latency_50 | s | 50th percentile latency of the gRPC get_sorted_shuffle_data process |
| get_sorted_shuffle_data_latency_90 | s | 90th percentile latency of the gRPC get_sorted_shuffle_data process |
| get_sorted_shuffle_data_latency_99 | s | 99th percentile latency of the gRPC get_sorted_shuffle_data process |
| get_sorted_shuffle_data_latency_sum | s | Total latency of the gRPC get_sorted_shuffle_data process |
gRPC Process number of latencies | send_shuffle_data | Count | Total number of the gRPC send_shuffle_data processes |
| get_local_shuffle_data | Count | Total number of the gRPC get_local_shuffle_data processes |
| get_memory_shuffle_data | Count | Total number of the gRPC get_memory_shuffle_data processes |
| get_shuffle_result_for_multi_part | Count | Total number of the gRPC get_shuffle_result_for_multi_part processes |
| report_shuffle_result | Count | Total number of the gRPC report_shuffle_result processes |
| get_sorted_shuffle_data | Count | Total number of the gRPC get_sorted_shuffle_data processes |
gRPC number of requests | open | Count | Number of open gRPC requests |
| send_shuffle_data_request_total | Count | Total number of netty send_shuffle_data_requests |
| get_local_shuffle_index_request_total | Count | Total count of netty get_local_shuffle_index_request |
| get_memory_shuffle_data_request_total | Count | Total count of netty get_memory_shuffle_data_request |
| get_sorted_shuffle_data_request_total | Count | Total count of netty get_sorted_shuffle_data_request |
gRPC Transport latency | send_shuffle_data_latency_50 | s | 50th percentile latency of grpc send_shuffle_data transport |
| send_shuffle_data_latency_90 | s | 90th percentile latency of grpc send_shuffle_data transport |
| send_shuffle_data_latency_99 | s | 99th percentile latency of grpc send_shuffle_data transport |
| send_shuffle_data_latency_sum | s | Total latency of grpc send_shuffle_data transport |
| get_local_shuffle_data_latency_50 | s | 50th percentile latency of grpc get_local_shuffle_data transport |
| get_local_shuffle_data_latency_90 | s | 90th percentile latency of grpc get_local_shuffle_data transport |
| get_local_shuffle_data_latency_99 | s | 99th percentile latency of grpc get_local_shuffle_data transport |
| get_local_shuffle_data_latency_sum | s | Total latency of grpc get_local_shuffle_data transport |
| get_memory_shuffle_data_latency_50 | s | 50th percentile latency of grpc get_memory_shuffle_data transport |
| get_memory_shuffle_data_latency_90 | s | 90th percentile latency of grpc get_memory_shuffle_data transport |
| get_memory_shuffle_data_latency_99 | s | 99th percentile latency of grpc get_memory_shuffle_data transport |
| get_memory_shuffle_data_latency_sum | s | Total latency of grpc get_memory_shuffle_data transport |
| get_sorted_shuffle_data_latency_50 | s | 50th percentile latency of grpc get_sorted_shuffle_data transport |
| get_sorted_shuffle_data_latency_90 | s | 90th percentile latency of grpc get_sorted_shuffle_data transport |
| get_sorted_shuffle_data_latency_99 | s | 99th percentile latency of grpc get_sorted_shuffle_data transport |
| get_sorted_shuffle_data_latency_sum | s | Total latency of grpc get_sorted_shuffle_data transport |
gRPC Transport latency count | send_shuffle_data | Count | Total count of grpc send_shuffle_data transport |
| get_local_shuffle_data | Count | Total count of grpc get_local_shuffle_data transport |
| get_memory_shuffle_data | Count | Total count of grpc get_memory_shuffle_data transport |
| get_sorted_shuffle_data | Count | Total count of grpc get_sorted_shuffle_data transport |
Netty connections | active_connection | Count | Number of active netty connections |
Netty exceptions | handle_exception | Count | Number of netty handle exceptions |
Netty Process latency | send_shuffle_data_request_latency_50 | s | 50th percentile latency of the netty send_shuffle_data_request process |
| send_shuffle_data_request_latency_90 | s | 90th percentile latency of the netty send_shuffle_data_request process |
| send_shuffle_data_request_latency_99 | s | 99th percentile latency of the netty send_shuffle_data_request process |
| send_shuffle_data_request_latency_sum | s | Total latency of the netty send_shuffle_data_request process |
| get_local_shuffle_data_request_latency_50 | s | 50th percentile latency of the netty get_local_shuffle_data_request process |
| get_local_shuffle_data_request_latency_90 | s | 90th percentile latency of the netty get_local_shuffle_data_request process |
| get_local_shuffle_data_request_latency_99 | s | 99th percentile latency of the netty get_local_shuffle_data_request process |
| get_local_shuffle_data_request_latency_sum | s | Total latency of the netty get_local_shuffle_data_request process |
| get_local_shuffle_index_request_latency_50 | s | 50th percentile latency of the netty get_local_shuffle_index_request process |
| get_local_shuffle_index_request_latency_90 | s | 90th percentile latency of the netty get_local_shuffle_index_request process |
| get_local_shuffle_index_request_latency_99 | s | 99th percentile latency of the netty get_local_shuffle_index_request process |
| get_local_shuffle_index_request_latency_sum | s | Total latency of the netty get_local_shuffle_index_request process |
| get_memory_shuffle_data_request_latency_50 | s | 50th percentile latency of the netty get_memory_shuffle_data_request process |
| get_memory_shuffle_data_request_latency_90 | s | 90th percentile latency of the netty get_memory_shuffle_data_request process |
| get_memory_shuffle_data_request_latency_99 | s | 99th percentile latency of the netty get_memory_shuffle_data_request process |
| get_memory_shuffle_data_request_latency_sum | s | Total latency of the netty get_memory_shuffle_data_request process |
| get_sorted_shuffle_data_request_latency_50 | s | 50th percentile latency of the netty get_sorted_shuffle_data_request process |
| get_sorted_shuffle_data_request_latency_90 | s | 90th percentile latency of the netty get_sorted_shuffle_data_request process |
| get_sorted_shuffle_data_request_latency_99 | s | 99th percentile latency of the netty get_sorted_shuffle_data_request process |
| get_sorted_shuffle_data_request_latency_sum | s | Total latency of the netty get_sorted_shuffle_data_request process |
Netty Process number of latencies | send_shuffle_data_request | Count | Total number of netty send_shuffle_data_request processes |
| get_local_shuffle_data_request | Count | number of netty get_local_shuffle_data_request processes |
| get_local_shuffle_index_request | Count | Total number of netty get_local_shuffle_index_request processes |
| get_memory_shuffle_data_request | Count | Total number of netty get_memory_shuffle_data_request processes |
| get_sorted_shuffle_data_request | Count | Total number of netty get_sorted_shuffle_data_request processes |
Netty Transport latency | send_shuffle_data_request_latency_50 | s | 50th percentile latency of netty send_shuffle_data_request transport |
| send_shuffle_data_request_latency_90 | s | 90th percentile latency of netty send_shuffle_data_request transport |
| send_shuffle_data_request_latency_99 | s | 99th percentile latency of netty send_shuffle_data_request transport |
| send_shuffle_data_request_latency_sum | s | Total latency of netty send_shuffle_data_request transport |
| get_local_shuffle_data_request_latency_50 | s | 50th percentile latency of netty get_local_shuffle_data_request transport |
| get_local_shuffle_data_request_latency_90 | s | 90th percentile latency of netty get_local_shuffle_data_request transport |
| get_local_shuffle_data_request_latency_99 | s | 99th percentile latency of netty get_local_shuffle_data_request transport |
| get_local_shuffle_data_request_latency_sum | s | Total latency of netty get_local_shuffle_data_request transport |
| get_local_shuffle_index_request_latency_50 | s | 50th percentile latency of netty get_local_shuffle_index_request transport |
| get_local_shuffle_index_request_latency_90 | s | 90th percentile latency of netty get_local_shuffle_index_request transport |
| get_local_shuffle_index_request_latency_99 | s | 99th percentile latency of netty get_local_shuffle_index_request transport |
| get_local_shuffle_index_request_latency_sum | s | Total latency of netty get_local_shuffle_index_request transport |
| get_memory_shuffle_data_request_latency_50 | s | 50th percentile latency of netty get_memory_shuffle_data_request transport |
| get_memory_shuffle_data_request_latency_90 | s | 90th percentile latency of netty get_memory_shuffle_data_request transport |
| get_memory_shuffle_data_request_latency_99 | s | 99th percentile latency of netty get_memory_shuffle_data_request transport |
| get_memory_shuffle_data_request_latency_sum | s | Total latency of netty get_memory_shuffle_data_request transport |
| get_sorted_shuffle_data_request_latency_50 | s | 50th percentile latency of netty get_sorted_shuffle_data_request transport |
| get_sorted_shuffle_data_request_latency_90 | s | 90th percentile latency of netty get_sorted_shuffle_data_request transport |
| get_sorted_shuffle_data_request_latency_99 | s | 99th percentile latency of netty get_sorted_shuffle_data_request transport |
| get_sorted_shuffle_data_request_latency_sum | s | Total latency of netty get_sorted_shuffle_data_request transport |
Netty Transport number of latencies | send_shuffle_data_request | Count | Total number of netty send_shuffle_data_request transports |
| get_local_shuffle_data_request | Count | Total number of netty get_local_shuffle_data_request transports |
| get_local_shuffle_index_request | Count | Number of netty get_local_shuffle_index_request transports |
| get_memory_shuffle_data_request | Count | Total number of netty get_memory_shuffle_data_request transports |
| get_sorted_shuffle_data_request | Count | Total number of netty get_sorted_shuffle_data_request transports |
Health Status | is_healthy | Count | Whether the current node is healthy |
number of Partitions | partition_num_with_node | Count | Number of partitions currently storing data on the ShuffleServer |
| total_partition_num | Count | Number of partitions stored on the node |
| total_huge_partition_num | Count | Total number of huge partitions on the node |
| huge_partition_num | Count | Current number of huge partitions on the node |
Pending tasks | pending_tasks_num_for_boss_group | Count | Number of pending tasks in the netty boss group |
| pending_tasks_num_for_worker_group | Count | Number of pending tasks in the netty worker group |
Read data volume | total_read_data | Bytes | Total data volume read by all applications from the ShuffleServer, including both data and index data |
| total_read_local_data_file | Bytes | Data volume read from the ShuffleServer |
| total_read_local_index_file | Bytes | index data volume read from the ShuffleServer |
| total_read_memory_data | Bytes | Data volume read from memory by the ShuffleServer |
| read_local_data_file_buffer_size | Bytes | Current buffer size for reading local storage data |
| read_local_index_file_buffer_size | Bytes | Current buffer size for reading local storage index |
| read_memory_data_buffer_size | Bytes | Current buffer size for reading memory data |
Remove App resource count | total_time_count | Count | Total number of APP resource removal operations |
| total_by_shuffle_ids_time_count | Count | Total number of operations for removing partial shuffleIds resources of an APP |
Remove App resource time consumption | total_time_50 | s | 50th percentile time consumption for removing APP resources |
| total_time_90 | s | 90th percentile time consumption for removing APP resources |
| total_time_99 | s | 99th percentile time consumption for removing APP resources |
| total_time_sum | s | Total time consumption for removing APP resources |
| total_by_shuffle_ids_time_50 | s | 50th percentile time consumption for removing partial shuffleIds resources of an APP |
| total_by_shuffle_ids_time_90 | s | 90th percentile time consumption for removing partial shuffleIds resources of an APP |
| total_by_shuffle_ids_time_99 | s | 99th percentile time consumption for removing partial shuffleIds resources of an APP |
| total_by_shuffle_ids_time_sum | s | Total time consumption for removing partial shuffleIds resources of an APP |
Memory request | total_require_read_memory_num | Count | Total number of memory requests for reading data |
| total_require_read_memory_retry_num | Count | Total number of retry requests for reading data memory allocation |
| total_require_read_memory_failed_num | Count | Total number of failed memory requests for reading data |
Buffer request | total_failed_for_huge_partition | Count | Number of failed huge memory requests from shuffle client to shuffle server, where huge memory is defined by rss.server.huge-partition.size.threshold. |
| total_failed_for_regular_partition | Count | Number of failed memory requests from shuffle client to shuffle server for storing shuffle data |
Storage | storage_used_bytes | Bytes | Storage usage size |
| local_storage_total_space | Bytes | Total local storage size of the current node |
| local_storage_whole_disk_used_space | Bytes | Used local storage space size of the current node |
| local_storage_whole_disk_free_space | Bytes | Free local storage space size of the current node |
| local_storage_service_used_space | Bytes | Local storage space size used by this service on the current node. |
Storage Directory | local_storage_total_dirs_num | Count | Number of available local storage root directories on the current node |
| local_storage_corrupted_dirs_num | Count | Number of unavailable local storage root directories on the current node |
Storage ratio | local_storage_used_space_ratio | % | Local storage space utilization ratio of the current node |
number of Applications | total_app_not_found_num | Count | Number of appNotFound occurrences |
| total_app_num | Count | Total number of all applications processed on the node |
| total_app_with_huge_partition_num | Count | Total number of applications with huge partitions |
| app_with_huge_partition_num | Count | Cumulative value of applications with huge partitions on the node |
gRPC thread pool | blocking_queue_size | Count | Number of queued tasks in the grpc thread pool queue |
| server_connection_number | Count | Current number of grpc connections |
Number of Write Blocks | total_write_block | Count | Number of blocks flushed by the ShuffleServer to disk/remote storage |
number of writes | handler_write_time_summary_count | Count | Number of times the ShuffleServer flushes data to disk/remote storage |
Write exception | total_write_exception | Count | Number of failed flushes to Local/remote storage |
Write threads | total_write_handler | Count | Number of concurrent data Flush threads in the ShuffleServer |
Slow Writes | total_write_slow | Count | Number of slow flushes to Local/remote storage (threshold controlled by rss.server.write.slow.threshold) |
Write latency | total_write_time | ms | Time consumed by the ShuffleServer to Flush data to disk/remote storage |
| handler_write_time_summary_50 | ms | 50th percentile latency of ShuffleServer Flush to disk/remote storage |
| handler_write_time_summary_90 | ms | 90th percentile latency of ShuffleServer Flush to disk/remote storage |
| handler_write_time_summary_99 | ms | 99th percentile latency of ShuffleServer Flush to disk/remote storage |
| handler_write_time_summary_sum | ms | Total time consumed by the ShuffleServer to Flush data to disk/remote storage |
| total_write_lock_duration | ms | Total lock duration for LocalFileWrite |
Process Cpu Load | ProcessCpuLoad | % | CPU utilization |
GC count | YGC | Count | Young GC count |
| FGC | Count | Full GC count |
GC time | FGCT | s | Full GC consumption time |
| GCT | s | Garbage collection consumption time |
| YGCT | s | Young GC consumption time |
Memory area percentage | S0 | % | Percentage of used Survivor 0 memory |
| E | % | Percentage of used Eden memory |
| CCS | % | Percentage of used compressed class space memory |
| S1 | % | Percentage of used Survivor 1 memory |
| O | % | Percentage of used Old memory |
| M | % | Percentage of used Metaspace memory |
JVM Memory | MemHeapUsedM | MBytes | Used heap memory |
| MemNonHeapUsedM | MBytes | Used non-heap memory |
| MemHeapCommittedM | MBytes | Committed heap memory |
| MemNonHeapCommittedM | MBytes | Committed non-heap memory |
| MemHeapMaxM | MBytes | Maximum heap memory |
| MemHeapInitM | MBytes | Initial heap memory |
| MemNonHeapInitM | MBytes | Initial non-heap memory |
File Descriptors | MaxFileDescriptorCount | Count | Maximum number of file descriptors |
| OpenFileDescriptorCount | Count | Number of open file descriptors |
JVM Threads | PeakThreadCount | Count | Peak number of threads |
| ThreadCount | Count | Total number of threads |
| DaemonThreadCount | Count | Daemon number of threads |
피드백