Title | Metric Name | Metric Unit | Metric Meaning |
Number of servers | Running | count | Number of Shuffle Servers registered to the coordinator |
| Excluded | count | Number of ShuffleServer nodes in the exclude file list |
| Unhealthy | count | Number of unhealthy ShuffleServer nodes |
number of Applications | Running | count | Number of running applications registered to the coordinator |
| Total | count | Total number of applications |
Number of requests | LoadDenied | count | Number of requests denied due to excessive memory load on the ShuffleServer |
| QuotaDenied | count | Number of requests denied due to user-level Application quota quantity control |
GC count | YGC | count | Young GC count |
| FGC | count | Full GC count |
GC time | FGCT | s | Full GC consumption time |
| GCT | s | Garbage collection consumption time |
| YGCT | s | Young GC consumption time |
Memory zone proportion | S0 | % | Percentage of used Survivor 0 memory |
| E | % | Percentage of used Eden memory |
| CCS | % | Percentage of used compressed class space memory |
| S1 | % | Percentage of used Survivor 1 memory |
| O | % | Percentage of used Old memory |
| M | % | Percentage of used Metaspace memory |
JVM memory | MemHeapUsedM | bytes | Used heap memory |
| MemNonHeapUsedM | bytes | Used non-heap memory |
| MemHeapCommittedM | bytes | Committed heap memory |
| MemNonHeapCommittedM | bytes | Committed non-heap memory |
| MemHeapMaxM | bytes | Maximum heap memory |
| MemNonHeapMaxM | bytes | Maximum non-heap memory |
| MemHeapInitM | bytes | Initial heap memory |
| MemNonHeapInitM | bytes | Initial non-heap memory |
Number of file descriptors | MaxFileDescriptorCount | count | Maximum number of file descriptors |
| OpenFileDescriptorCount | count | Number of open file descriptors |
Number of processes | PeakThreadCount | count | Peak number of threads |
| ThreadCount | count | Total number of threads |
| DaemonThreadCount | count | Daemon number of threads |
Process runtime | Uptime | s | Process runtime |
Process start time. | StartTime | s | Process start time. |
Heap memory utilization | MemHeapUsedRate | % | Heap memory usage percentage |
Title | Metric Name | Metric Unit | Metric Meaning |
Data Info | Received | bytes | Amount of data received by the ShuffleServer |
| Write | bytes | Amount of data flushed by the ShuffleServer to disk/remote storage |
| WriteHdfs | bytes | Amount of data written to HDFS |
| WriteLocal | bytes | Amount of data written to local storage |
Number of Write Blocks | Count | count | Number of blocks flushed by the ShuffleServer to disk/remote storage |
Write Flush Time | Time | ms | Time consumed by the ShuffleServer to Flush data to disk/remote storage |
Write Flush threads | Count | count | Number of concurrent threads flushing data to disk in the ShuffleServer |
Write Flushes Number | Total | count | Number of times the ShuffleServer flushes data to disk/remote storage |
| Slow | count | Number of slow flushes to disk/remote storage by the ShuffleServer, where the time threshold is determined by rss.server.write.slow.threshold. |
Number of applications | App | count | Number of applications currently storing data on the ShuffleServer |
Number of partitions | Partition | count | Number of partitions currently storing data on the ShuffleServer |
Event status | Queue | count | Number of events currently in the ShuffleServer flush queue |
| FailedWriten | count | Number of events that failed to be persisted |
| Dropped | count | Number of events dropped due to non-existent application or flush timeout |
Data Read | Total | bytes | Total data volume read by all applications from the ShuffleServer, including both data and index data |
| DataFromLocalStorage | bytes | Data volume read from the ShuffleServer |
| IndexFromLocalStorage | bytes | index data volume read from the ShuffleServer |
| DataFromMemory | bytes | Data volume read from memory by the ShuffleServer |
Read time | Time | ms | Total read duration |
Memory request | Total | bytes | Total memory requested for reading data |
| Retry | bytes | Total memory requested for retries during data read requests |
| Failed | bytes | Amount of memory that failed to be allocated for data read requests |
Node health status | IsHealthy | count | Whether the current node is healthy |
Buffer status | Allocated | bytes | Amount of memory allocated for storing data |
| InFlush | bytes | Amount of memory pending flush to disk |
| Used | bytes | The amount of memory used to store shuffle data, with its upper limit determined by rss.server.buffer.capacity. |
| ReadUsed | bytes | The amount of memory used to cache shuffle read data, with its upper limit determined by rss.server.read.buffer.capacity. |
Local storage path | Total | count | Number of available local storage root directories on the current node |
| Corrupted | count | Number of unavailable local storage root directories on the current node |
Local storage capacity | Total | bytes | Total local storage size of the current node |
| Used | bytes | Used local storage space size of the current node |
Local storage utilization | Ratio | % | Local storage space utilization ratio of the current node |
Local storage write statistics | Total | count | Total number of writes to local storage |
| Retry | count | Number of retried writes to local storage |
| Failed | count | Number of failed writes to local storage |
| Success | count | Number of successful writes to local storage |
App | Total | count | Cumulative number of applications processed on the node |
| TotalWithHugePartition | count | Cumulative number of applications with huge partitions |
| WithHugePartition | count | Number of applications with huge partitions on the node |
Partition | Total | count | Cumulative number of partitions stored on the node |
| TotalHuge | count | Cumulative number of huge partitions on the node |
| Huge | count | Current number of huge partitions on the node |
Buffer failure statistics | Huge | count | Number of failed huge memory requests from shuffle client to ShuffleServer, where huge memory is defined by rss.server.huge-partition.size.threshold. |
| Regular | count | Number of failed memory requests from shuffle client to ShuffleServer for storing shuffle data |
JVM memory | MemHeapUsedM | bytes | Used heap memory |
| MemNonHeapUsedM | bytes | Used non-heap memory |
| MemHeapCommittedM | bytes | Committed heap memory |
| MemNonHeapCommittedM | bytes | Committed non-heap memory |
| MemHeapMaxM | bytes | Maximum heap memory |
| MemNonHeapMaxM | bytes | Maximum non-heap memory |
| MemHeapInitM | bytes | Initial heap memory |
| MemNonHeapInitM | bytes | Initial non-heap memory |
Number of file descriptors | MaxFileDescriptorCount | count | Maximum number of file descriptors |
| OpenFileDescriptorCount | count | Number of open file descriptors |
number of processes | PeakThreadCount | count | Peak number of threads |
| ThreadCount | count | Total number of threads |
| DaemonThreadCount | count | Daemon thread count |
Process runtime | Uptime | s | Process runtime |
Process start time. | StartTime | s | Process start time. |
GC count | YGC | count | Young GC count |
| FGC | | Full GC count |
GC time | FGCT | s | Full GC consumption time |
| GCT | | Garbage collection consumption time |
| YGCT | | Young GC consumption time |
Memory zone proportion | S0 | % | Percentage of used Survivor 0 memory |
| E | | Percentage of used Eden memory |
| CCS | | Percentage of used compressed class space memory |
| S1 | | Percentage of used Survivor 1 memory |
| O | | Percentage of used Old memory |
| M | | Percentage of used Metaspace memory |
Heap memory utilization | MemHeapUsedRate | % | Heap memory usage percentage |
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback