\page icelakesp Intel® Icelake SP
This page is valid for Icelake SP.
Available performance monitors for the Intel® Icelake SP microarchitecture
- \ref ICX_FIXED "Fixed-purpose counters"
- \ref ICX_METRICS "Performance metric counters"
- \ref ICX_PMC "General-purpose counters"
- \ref ICX_THERMAL "Thermal counters"
- \ref ICX_VOLTAGE "Core voltage counters"
- \ref ICX_POWER "Power measurement counters"
- \ref ICX_UBOX "Uncore global counters"
- \ref ICX_CBOX "Last level cache counters"
- \ref ICX_MBOX "Memory channel counters"
- \ref ICX_MBOXFREERUN "Memory controller counters"
- \ref ICX_WBOX "Power control unit counters"
- \ref ICX_QBOX "UPI interface counters"
- \ref ICX_SBOX "Mesh-to-UPI counters (M3UPI)"
- \ref ICX_IBOX "IIO box counters"
- \ref ICX_BBOX "Mesh2Memory counters"
- \ref ICX_PBOX "Mesh2PCIe counters"
Counters available for each hardware thread
\anchor ICX_FIXED
Fixed-purpose counters
Since the Core2 microarchitecture, Intel® provides a set of fixed-purpose counters. Each can measure only one specific event. The Intel® Icelake SP architecture adds a fourth fixed-purpose counter for the event TOPDOWN_SLOTS.
Counter and events
| Counter name |
Event name |
| FIXC0 |
INSTR_RETIRED_ANY |
| FIXC1 |
CPU_CLK_UNHALTED_CORE |
| FIXC2 |
CPU_CLK_UNHALTED_REF |
| FIXC3 |
TOPDOWN_SLOTS |
Available Options
| Option |
Argument |
Description |
Comment |
| anythread |
N |
Set bit 2+(index*4) in config register |
|
| kernel |
N |
Set bit (index*4) in config register |
|
\anchor ICX_METRICS
Performance metric counters
With the Intel® Icelake SP microarchitecture a new class of core-local counters was introduced, the so-called perf-metrics. The reflect the first level of the Top-down Microarchitecture Analysis tree. The events return the fraction of slots used by the event.
Counter and events
| Counter name |
Event name |
| TMA0 |
RETIRING |
| TMA1 |
BAD_SPECULATION |
| TMA2 |
FRONTEND_BOUND |
| TMA3 |
BACKEND_BOUND |
\anchor ICX_PMC
General-purpose counters
Commonly the Intel® Icelake SP microarchitecture provides 4 general-purpose counters consisting of a config and a counter register.
Counter and events
| Counter name |
Event name |
| PMC0 |
* |
| PMC1 |
* |
| PMC2 |
* |
| PMC3 |
* |
Available Options
| Option |
Argument |
Description |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| kernel |
N |
Set bit 17 in config register |
|
| anythread |
N |
Set bit 21 in config register |
The anythread option is deprecated but still supported. Please check the official Intel® documentation how to use the option properly. |
| threshold |
8 bit hex value |
Set bits 24-31 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| in_transaction |
N |
Set bit 32 in config register |
Only available if Intel® Transactional Synchronization Extensions are available |
| in_transaction_aborted |
N |
Set bit 33 in config register |
Only counter PMC2 and only if Intel® Transactional Synchronization Extensions are available |
If HyperThreading is disabled, each core can use 8 general-purpose counters named <4-7>.
\anchor ICX_THERMAL
Thermal counters
The Intel® Icelake SP microarchitecture provides one register for the current core temperature.
Counter and events
| Counter name |
Event name |
| TMP0 |
TEMP_CORE |
\anchor ICX_VOLTAGE
Core voltage counters
The Intel® Icelake SP microarchitecture provides one register for the current core voltage.
Counter and events
| Counter name |
Event name |
| VTG0 |
VOLTAGE_CORE |
Counters available for one hardware thread per socket
\anchor ICX_POWER
Power counter
The Intel® Icelake SP microarchitecture provides measurements of the current power consumption through the RAPL interface.
Counter and events
| Counter name |
Event name |
| PWR0 |
PWR_PKG_ENERGY |
| PWR1 |
PWR_PP0_ENERGY |
| PWR2 |
PWR_PP1_ENERGY |
| PWR3 |
PWR_DRAM_ENERGY |
| PWR4 |
PWR_PLATFORM_ENERGY |
\anchor ICX_UBOX
Uncore global counters
The Intel® Icelake SP microarchitecture provides measurements for the global uncore.
Counter and events
| Counter name |
Event name |
| UBOX0 |
* |
| UBOX1 |
* |
| UBOXFIX |
UNCORE_CLOCK |
Available Options
| Option |
Argument |
Description |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| threshold |
8 bit hex value |
Set bits 24-31 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
\anchor ICX_CBOX
Last level cache counters
The Intel® Icelake SP microarchitecture provides measurements for the last level cache segments.
Counter and events
| Counter name |
Event name |
| CBOX<0-39>C0 |
* |
| CBOX<0-39>C1 |
* |
| CBOX<0-39>C2 |
* |
| CBOX<0-39>C3 |
* |
Available Options
| Option |
Argument |
Description |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| threshold |
8 bit hex value |
Set bits 24-28 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| state |
8 bit hex value |
Set bits 8-15 in config register. Similar to umask |
LLC F: 0x80, LLC M: 0x40, LLC E: 0x20, LLC S: 0x10, SF H: 0x08, SF E: 0x04, SF S: 0x02, LLC I: 0x01. Only for the LLC_LOOKUP events |
| match0 |
28 bit hex value |
Set bits 32-57 in config register named UmaskExt |
See the special event handling section and 3rd Gen Intel® Xeon® Processor Scalable Family, Codename Ice Lake, Uncore Performance Monitoring Reference Manual for more information. |
Special event handling
Event LLC_LOOKUP uses the umask field for the state specification. Further filters are in a field called UmaskExt. These filters can be addressed with the MASK0 option, thus LLC_LOOKUP_I:CBOX0C0:MATCH0=0x8 would count cache lines in I (=invalid) state filtered by RFOs (MATCH0=0x8). Most LLC_LOOKUP events use umask/state 0xFF for all states. If you want to use multiple states, use the STATE option like STATE=0xE for all SF states.
The events TOR_INSERTS and TOR_OCCUPANCY also use the UmaskExt field but with different width and meaning:
| Bit offset |
Description |
| 0 |
Data Reads- local or remote. includes prefetches |
| 1 |
All write transactions to the LLC - including writebacks to LLC and uncacheable write transactions
Does not include evict cleans or invalidates |
| 2 |
Flush or Invalidates |
| 3 |
RFOs - local or remote. includes prefetches |
| 4 |
Code Reads- local or remote. includes prefetches |
| 5 |
Any local or remote transaction to the LLC. Includes prefetches |
| 6 |
Any local transaction to LLC, including prefetches from Core |
| 7 |
Any local prefetch to LLC from an LLC |
| 8 |
Any local prefetch to LLC from Core |
| 9 |
Snoop transactions to the LLC from a remote agent |
| 10 |
Non-snoop transactions to the LLC from a remote agent |
| 11 |
Transactions to locally homed addresses |
| 12 |
Transactions to remotely homed addresses |
The events WRITE_NO_CREDITS and READ_NO_CREDITS use the UmaskExt as real extension of the default umask field. Each of the bits corresponds to a memory controller (0-13). The first eight are covered by the bits in umask. The other 6 bits are in the UmaskExt field addressable with the MATCH0 option.
The event LLC_VICTIMS uses the MATCH0 option to differentiate between 'local only' and 'remote only' victims. If nothing is set, 'all' are counted. There are only two settings: MATCH0=0x20 for 'local only' and MATCH0=0x80 for 'remote only'.
\anchor ICX_MBOX
Memory channel counters
The Intel® Icelake SP microarchitecture provides measurements of the memory channels of each integrated Memory Controllers (iMC) in the Uncore. The description from Intel®:
The Ice Lake integrated Memory Controller provides the interface to DRAM and communicates to the rest of the Uncore through the Mesh2Mem block.
The memory controller also provides a variety of RAS features, such as ECC, lockstep, memory access retry, memory scrubbing, thermal throttling, mirroring, and rank sparing.
The integrated Memory Controllers performance counters are exposed to the operating system through MMIO interfaces. There may be four memory controllers in the system. Each controller provides two memory channels. Each channel has 4 different general-purpose counters and one fixed counter for the DRAM clock. The channels of the first memory controller are MBOX0-1, the two channels of the second memory controller are named MBOX2-3, and so on. The name MBOX originates from the Nehalem EX Uncore monitoring where those functional units are called MBOX.
Counter and events
| Counter name |
Event name |
| MBOX<0-7>C0 |
* |
| MBOX<0-7>C1 |
* |
| MBOX<0-7>C2 |
* |
| MBOX<0-7>C3 |
* |
| MBOX<0-7>FIX |
DRAM_CLOCKTICKS |
Available Options (Only for counter MBOX<0-7>C<0-3>)
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
8 bit hex value |
Set bits 24-31 in config register |
|
\anchor ICX_MBOXFREERUN
Memory controller counters
The Intel® Icelake SP microarchitecture provides measurements of the integrated Memory Controllers (iMC) in the Uncore. The description from Intel®:
The Ice Lake integrated Memory Controller provides the interface to DRAM and communicates to the rest of the Uncore through the Mesh2Mem block.
The memory controller also provides a variety of RAS features, such as ECC, lockstep, memory access retry, memory scrubbing, thermal throttling, mirroring, and rank sparing.
Each memory controller provides five free-running counters.
Counter and events
| Counter name |
Event name |
| MDEV<0-3>C0 |
DDR_READ_BYTES |
| MDEV<0-3>C1 |
DDR_WRITE_BYTES |
| MDEV<0-3>C2 |
PMM_READ_BYTES |
| MDEV<0-3>C3 |
PMM_WRITE_BYTES |
| MDEV<0-3>C4 |
IMC_DEV_CLOCKTICKS |
Info: Despite the event names contain the unit BYTES in them, they are counting in a cache line granularity
Mapping between MBOX<0-7>C<0-3> and MDEV<0-3>C<0-4>:
| MDEV |
MBOX |
| MDEV0 |
MBOX<0-1> |
| MDEV1 |
MBOX<2-3> |
| MDEV2 |
MBOX<4-5> |
| MDEV3 |
MBOX<6-7> |
\anchor ICX_WBOX
Power control unit counters
The Intel® Icelake SP microarchitecture provides measurements of the power control unit (PCU) in the Uncore. The description from Intel®:
The PCU is the primary Power Controller for the Ice Lake die, responsible for distributing power to core/uncore components and thermal management. It runs in firmware on an internal micro-controller and coordinates the socket’s power states.
The PCU performance counters are exposed to the operating system through the MSR interface. The name WBOX originates from the Nehalem EX Uncore monitoring where those functional units are called WBOX.
Counter and events
| Counter name |
Event name |
| WBOX0 |
* |
| WBOX1 |
* |
| WBOX2 |
* |
| WBOX3 |
* |
| WBOX0FIX |
CORES_IN_C3 |
| WBOX1FIX |
CORES_IN_C6 |
| WBOX2FIX |
CORES_IN_P3 |
| WBOX3FIX |
CORES_IN_P6 |
Available Options (Only for WBOX<0-3> counters)
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
5 bit hex value |
Set bits 24-28 in config register |
|
| occupancy_edgedetect |
N |
Set bit 31 in config register |
|
| occupancy_invert |
N |
Set bit 30 in config register |
|
\anchor ICX_QBOX
UPI interface counters
The Intel® Skylake SP microarchitecture provides measurements of the Ultra Path Interconnect Link layer (UPI) in the Uncore. The description from Intel®:
Snow Ridge uses a coherent interconnect for scaling to multiple sockets known as
Intel® Ultra Path Interconnect (UPI). Intel® UPI technology provides a cache coherent
socket to socket external communication interface between processors. Intel® UPI is
also used as a coherent communication interface between processors and OEM 3rd
party Node Controllers (XNC).
The UPI hardware performance counters are exposed to the operating system through PCI interfaces. There are three of those interfaces for the UPI. The actual amount of QBOX counters depend on the CPU core count of one socket. If your system has not all interfaces but interface 0 does not work, try the other ones. The QBOX was introduced for the Skylake microarchitecture.
Counter and events
| Counter name |
Event name |
| QBOX<0-2>C0 |
* |
| QBOX<0-2>C1 |
* |
| QBOX<0-2>C2 |
* |
| QBOX<0-2>C3 |
* |
Available Options
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
8 bit hex value |
Set bits 24-31 in config register |
|
\anchor ICX_SBOX
Mesh-to-UPI counters (M3UPI)
The Intel® Skylake SP microarchitecture provides measurements of the Mesh-to-UPI (M3UPI) interface in the Uncore. The description from Intel®:
M3UPI is the interface between the mesh and the Intel® UPI Link Layer. It is
responsible for translating between mesh protocol packets and flits that are used for
transmitting data across the Intel® UPI interface. It performs credit checking between
the local Intel® UPI LL, the remote Intel® UPI LL and other agents on the local mesh.
The Mesh-to-UPI performance counters are exposed to the operating system through PCI interfaces. Since the SBOXes manage the traffic from the LLC-connecting mesh interface on the socket with the UPI interfaces (QBOXes), the amount is similar to the amount of QBOXes (3). See at QBOXes how many are available for which system configuration.
Counter and events
| Counter name |
Event name |
| SBOX<0-2>C0 |
* |
| SBOX<0-2>C1 |
* |
| SBOX<0-2>C2 |
* |
| SBOX<0-2>C3 |
* |
Available Options
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
8 bit hex value |
Set bits 24-31 in config register |
|
\anchor ICX_IBOX
IIO box counters
The Intel® Skylake SP microarchitecture provides measurements of the IIO box in the Uncore. The description from Intel®:
IIO stacks are responsible for managing traffic between the PCIe domain and the Mesh
domain. The IIO PMON block is situated near the IIO stack’s traffic controller capturing
traffic controller as well as PCIe root port information. The traffic controller is
responsible for translating traffic coming in from the Mesh (through M2PCIe) and
processed by IRP into the PCIe domain to IO agents such as CBDMA, DMA and PCIe.
The IIO box counters are exposed to the operating system through the MSR interface.
Counter and events
| Counter name |
Event name |
| TCBOX<0-5>C0 |
* |
| TCBOX<0-5>C1 |
* |
| TCBOX<0-5>C2 |
* |
| TCBOX<0-5>C3 |
* |
| IBOX<0-5>C0 |
* |
| IBOX<0-5>C1 |
* |
| IBOX<0-5>PORT0 |
IIO_BANDWIDTH_IN_PORT0 |
| IBOX<0-5>PORT1 |
IIO_BANDWIDTH_IN_PORT1 |
| IBOX<0-5>PORT2 |
IIO_BANDWIDTH_IN_PORT2 |
| IBOX<0-5>PORT3 |
IIO_BANDWIDTH_IN_PORT3 |
| IBOX<0-5>PORT4 |
IIO_BANDWIDTH_IN_PORT4 |
| IBOX<0-5>PORT5 |
IIO_BANDWIDTH_IN_PORT5 |
| IBOX<0-5>PORT6 |
IIO_BANDWIDTH_IN_PORT6 |
| IBOX<0-5>PORT7 |
IIO_BANDWIDTH_IN_PORT7 |
Available Options (Only for TCBOX<0-5> and I<0-5> counters)
\anchor ICX_BBOX
Mesh2Memory counters
The Intel® Icelake SP microarchitecture provides measurements of the mesh (M2M) which connects the cores with the Uncore devices. The description from Intel®:
For all Boxes that must communicate with the Mesh, there are a common set of events
to capture various kinds of information about traffic flowing through their connection to
the Mesh. The same encodings are used to request the mesh events in each box.
The M2M devices is first introduced in the Intel® Skylake SP microarchitecture. There was no suitable unit name for this, so LIKWID calls them simply M2M.
Counter and events
| Counter name |
Event name |
| M2M<0-3>C0 |
* |
| M2M<0-3>C1 |
* |
| M2M<0-3>C2 |
* |
| M2M<0-3>C3 |
* |
Available Options
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
5 bit hex value |
Set bits 24-28 in config register |
|
\anchor ICX_PBOX
Mesh2PCIe counters
The Intel® Icelake SP microarchitecture provides measurements of the mesh to the IIO devices. The description from Intel®:
M2PCIe blocks manage the interface between the Mesh and each IIO stack.
Counter and events
| Counter name |
Event name |
| P<0-5>C0 |
* |
| P<0-5>C1 |
* |
| P<0-5>C2 |
* |
| P<0-5>C3 |
* |
Available Options
| Option |
Argument |
Operation |
Comment |
| edgedetect |
N |
Set bit 18 in config register |
|
| invert |
N |
Set bit 23 in config register |
|
| threshold |
5 bit hex value |
Set bits 24-28 in config register |
|