\page icelakesp Intel® Icelake SP

This page is valid for Icelake SP.

Available performance monitors for the Intel® Icelake SP microarchitecture

Counters available for each hardware thread

\anchor ICX_FIXED

Fixed-purpose counters

Since the Core2 microarchitecture, Intel® provides a set of fixed-purpose counters. Each can measure only one specific event. The Intel® Icelake SP architecture adds a fourth fixed-purpose counter for the event TOPDOWN_SLOTS.

Counter and events

Counter name Event name
FIXC0 INSTR_RETIRED_ANY
FIXC1 CPU_CLK_UNHALTED_CORE
FIXC2 CPU_CLK_UNHALTED_REF
FIXC3 TOPDOWN_SLOTS

Available Options

Option Argument Description Comment
anythread N Set bit 2+(index*4) in config register
kernel N Set bit (index*4) in config register
\anchor ICX_METRICS

Performance metric counters

With the Intel® Icelake SP microarchitecture a new class of core-local counters was introduced, the so-called perf-metrics. The reflect the first level of the Top-down Microarchitecture Analysis tree. The events return the fraction of slots used by the event.

Counter and events

Counter name Event name
TMA0 RETIRING
TMA1 BAD_SPECULATION
TMA2 FRONTEND_BOUND
TMA3 BACKEND_BOUND
\anchor ICX_PMC

General-purpose counters

Commonly the Intel® Icelake SP microarchitecture provides 4 general-purpose counters consisting of a config and a counter register.

Counter and events

Counter name Event name
PMC0 *
PMC1 *
PMC2 *
PMC3 *

Available Options

Option Argument Description Comment
edgedetect N Set bit 18 in config register
kernel N Set bit 17 in config register
anythread N Set bit 21 in config register The anythread option is deprecated but still supported. Please check the official Intel® documentation how to use the option properly.
threshold 8 bit hex value Set bits 24-31 in config register
invert N Set bit 23 in config register
in_transaction N Set bit 32 in config register Only available if Intel® Transactional Synchronization Extensions are available
in_transaction_aborted N Set bit 33 in config register Only counter PMC2 and only if Intel® Transactional Synchronization Extensions are available

If HyperThreading is disabled, each core can use 8 general-purpose counters named <4-7>. \anchor ICX_THERMAL

Thermal counters

The Intel® Icelake SP microarchitecture provides one register for the current core temperature.

Counter and events

Counter name Event name
TMP0 TEMP_CORE
\anchor ICX_VOLTAGE

Core voltage counters

The Intel® Icelake SP microarchitecture provides one register for the current core voltage.

Counter and events

Counter name Event name
VTG0 VOLTAGE_CORE

Counters available for one hardware thread per socket

\anchor ICX_POWER

Power counter

The Intel® Icelake SP microarchitecture provides measurements of the current power consumption through the RAPL interface.

Counter and events

Counter name Event name
PWR0 PWR_PKG_ENERGY
PWR1 PWR_PP0_ENERGY
PWR2 PWR_PP1_ENERGY
PWR3 PWR_DRAM_ENERGY
PWR4 PWR_PLATFORM_ENERGY
\anchor ICX_UBOX

Uncore global counters

The Intel® Icelake SP microarchitecture provides measurements for the global uncore.

Counter and events

Counter name Event name
UBOX0 *
UBOX1 *
UBOXFIX UNCORE_CLOCK

Available Options

Option Argument Description Comment
edgedetect N Set bit 18 in config register
threshold 8 bit hex value Set bits 24-31 in config register
invert N Set bit 23 in config register
\anchor ICX_CBOX

Last level cache counters

The Intel® Icelake SP microarchitecture provides measurements for the last level cache segments.

Counter and events

Counter name Event name
CBOX<0-39>C0 *
CBOX<0-39>C1 *
CBOX<0-39>C2 *
CBOX<0-39>C3 *

Available Options

Option Argument Description Comment
edgedetect N Set bit 18 in config register
threshold 8 bit hex value Set bits 24-28 in config register
invert N Set bit 23 in config register
state 8 bit hex value Set bits 8-15 in config register. Similar to umask LLC F: 0x80, LLC M: 0x40, LLC E: 0x20, LLC S: 0x10, SF H: 0x08, SF E: 0x04, SF S: 0x02, LLC I: 0x01. Only for the LLC_LOOKUP events
match0 28 bit hex value Set bits 32-57 in config register named UmaskExt See the special event handling section and 3rd Gen Intel® Xeon® Processor Scalable Family, Codename Ice Lake, Uncore Performance Monitoring Reference Manual for more information.

Special event handling

Event LLC_LOOKUP uses the umask field for the state specification. Further filters are in a field called UmaskExt. These filters can be addressed with the MASK0 option, thus LLC_LOOKUP_I:CBOX0C0:MATCH0=0x8 would count cache lines in I (=invalid) state filtered by RFOs (MATCH0=0x8). Most LLC_LOOKUP events use umask/state 0xFF for all states. If you want to use multiple states, use the STATE option like STATE=0xE for all SF states.
The events TOR_INSERTS and TOR_OCCUPANCY also use the UmaskExt field but with different width and meaning:

Bit offset Description
0 Data Reads- local or remote. includes prefetches
1 All write transactions to the LLC - including writebacks to LLC and uncacheable write transactions Does not include evict cleans or invalidates
2 Flush or Invalidates
3 RFOs - local or remote. includes prefetches
4 Code Reads- local or remote. includes prefetches
5 Any local or remote transaction to the LLC. Includes prefetches
6 Any local transaction to LLC, including prefetches from Core
7 Any local prefetch to LLC from an LLC
8 Any local prefetch to LLC from Core
9 Snoop transactions to the LLC from a remote agent
10 Non-snoop transactions to the LLC from a remote agent
11 Transactions to locally homed addresses
12 Transactions to remotely homed addresses

The events WRITE_NO_CREDITS and READ_NO_CREDITS use the UmaskExt as real extension of the default umask field. Each of the bits corresponds to a memory controller (0-13). The first eight are covered by the bits in umask. The other 6 bits are in the UmaskExt field addressable with the MATCH0 option.

The event LLC_VICTIMS uses the MATCH0 option to differentiate between 'local only' and 'remote only' victims. If nothing is set, 'all' are counted. There are only two settings: MATCH0=0x20 for 'local only' and MATCH0=0x80 for 'remote only'.

\anchor ICX_MBOX

Memory channel counters

The Intel® Icelake SP microarchitecture provides measurements of the memory channels of each integrated Memory Controllers (iMC) in the Uncore. The description from Intel®:
The Ice Lake integrated Memory Controller provides the interface to DRAM and communicates to the rest of the Uncore through the Mesh2Mem block.
The memory controller also provides a variety of RAS features, such as ECC, lockstep, memory access retry, memory scrubbing, thermal throttling, mirroring, and rank sparing.

The integrated Memory Controllers performance counters are exposed to the operating system through MMIO interfaces. There may be four memory controllers in the system. Each controller provides two memory channels. Each channel has 4 different general-purpose counters and one fixed counter for the DRAM clock. The channels of the first memory controller are MBOX0-1, the two channels of the second memory controller are named MBOX2-3, and so on. The name MBOX originates from the Nehalem EX Uncore monitoring where those functional units are called MBOX.

Counter and events

Counter name Event name
MBOX<0-7>C0 *
MBOX<0-7>C1 *
MBOX<0-7>C2 *
MBOX<0-7>C3 *
MBOX<0-7>FIX DRAM_CLOCKTICKS

Available Options (Only for counter MBOX<0-7>C<0-3>)

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 8 bit hex value Set bits 24-31 in config register
\anchor ICX_MBOXFREERUN

Memory controller counters

The Intel® Icelake SP microarchitecture provides measurements of the integrated Memory Controllers (iMC) in the Uncore. The description from Intel®:
The Ice Lake integrated Memory Controller provides the interface to DRAM and communicates to the rest of the Uncore through the Mesh2Mem block.
The memory controller also provides a variety of RAS features, such as ECC, lockstep, memory access retry, memory scrubbing, thermal throttling, mirroring, and rank sparing.

Each memory controller provides five free-running counters.

Counter and events

Counter name Event name
MDEV<0-3>C0 DDR_READ_BYTES
MDEV<0-3>C1 DDR_WRITE_BYTES
MDEV<0-3>C2 PMM_READ_BYTES
MDEV<0-3>C3 PMM_WRITE_BYTES
MDEV<0-3>C4 IMC_DEV_CLOCKTICKS

Info: Despite the event names contain the unit BYTES in them, they are counting in a cache line granularity

Mapping between MBOX<0-7>C<0-3> and MDEV<0-3>C<0-4>:

MDEV MBOX
MDEV0 MBOX<0-1>
MDEV1 MBOX<2-3>
MDEV2 MBOX<4-5>
MDEV3 MBOX<6-7>
\anchor ICX_WBOX

Power control unit counters

The Intel® Icelake SP microarchitecture provides measurements of the power control unit (PCU) in the Uncore. The description from Intel®:
The PCU is the primary Power Controller for the Ice Lake die, responsible for distributing power to core/uncore components and thermal management. It runs in firmware on an internal micro-controller and coordinates the socket’s power states.
The PCU performance counters are exposed to the operating system through the MSR interface. The name WBOX originates from the Nehalem EX Uncore monitoring where those functional units are called WBOX.

Counter and events

Counter name Event name
WBOX0 *
WBOX1 *
WBOX2 *
WBOX3 *
WBOX0FIX CORES_IN_C3
WBOX1FIX CORES_IN_C6
WBOX2FIX CORES_IN_P3
WBOX3FIX CORES_IN_P6

Available Options (Only for WBOX<0-3> counters)

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 5 bit hex value Set bits 24-28 in config register
occupancy_edgedetect N Set bit 31 in config register
occupancy_invert N Set bit 30 in config register
\anchor ICX_QBOX

UPI interface counters

The Intel® Skylake SP microarchitecture provides measurements of the Ultra Path Interconnect Link layer (UPI) in the Uncore. The description from Intel®:
Snow Ridge uses a coherent interconnect for scaling to multiple sockets known as Intel® Ultra Path Interconnect (UPI). Intel® UPI technology provides a cache coherent socket to socket external communication interface between processors. Intel® UPI is also used as a coherent communication interface between processors and OEM 3rd party Node Controllers (XNC).
The UPI hardware performance counters are exposed to the operating system through PCI interfaces. There are three of those interfaces for the UPI. The actual amount of QBOX counters depend on the CPU core count of one socket. If your system has not all interfaces but interface 0 does not work, try the other ones. The QBOX was introduced for the Skylake microarchitecture.

Counter and events

Counter name Event name
QBOX<0-2>C0 *
QBOX<0-2>C1 *
QBOX<0-2>C2 *
QBOX<0-2>C3 *

Available Options

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 8 bit hex value Set bits 24-31 in config register
\anchor ICX_SBOX

Mesh-to-UPI counters (M3UPI)

The Intel® Skylake SP microarchitecture provides measurements of the Mesh-to-UPI (M3UPI) interface in the Uncore. The description from Intel®:
M3UPI is the interface between the mesh and the Intel® UPI Link Layer. It is responsible for translating between mesh protocol packets and flits that are used for transmitting data across the Intel® UPI interface. It performs credit checking between the local Intel® UPI LL, the remote Intel® UPI LL and other agents on the local mesh.
The Mesh-to-UPI performance counters are exposed to the operating system through PCI interfaces. Since the SBOXes manage the traffic from the LLC-connecting mesh interface on the socket with the UPI interfaces (QBOXes), the amount is similar to the amount of QBOXes (3). See at QBOXes how many are available for which system configuration.

Counter and events

Counter name Event name
SBOX<0-2>C0 *
SBOX<0-2>C1 *
SBOX<0-2>C2 *
SBOX<0-2>C3 *

Available Options

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 8 bit hex value Set bits 24-31 in config register
\anchor ICX_IBOX

IIO box counters

The Intel® Skylake SP microarchitecture provides measurements of the IIO box in the Uncore. The description from Intel®:
IIO stacks are responsible for managing traffic between the PCIe domain and the Mesh domain. The IIO PMON block is situated near the IIO stack’s traffic controller capturing traffic controller as well as PCIe root port information. The traffic controller is responsible for translating traffic coming in from the Mesh (through M2PCIe) and processed by IRP into the PCIe domain to IO agents such as CBDMA, DMA and PCIe.
The IIO box counters are exposed to the operating system through the MSR interface.

Counter and events

Counter name Event name
TCBOX<0-5>C0 *
TCBOX<0-5>C1 *
TCBOX<0-5>C2 *
TCBOX<0-5>C3 *
IBOX<0-5>C0 *
IBOX<0-5>C1 *
IBOX<0-5>PORT0 IIO_BANDWIDTH_IN_PORT0
IBOX<0-5>PORT1 IIO_BANDWIDTH_IN_PORT1
IBOX<0-5>PORT2 IIO_BANDWIDTH_IN_PORT2
IBOX<0-5>PORT3 IIO_BANDWIDTH_IN_PORT3
IBOX<0-5>PORT4 IIO_BANDWIDTH_IN_PORT4
IBOX<0-5>PORT5 IIO_BANDWIDTH_IN_PORT5
IBOX<0-5>PORT6 IIO_BANDWIDTH_IN_PORT6
IBOX<0-5>PORT7 IIO_BANDWIDTH_IN_PORT7

Available Options (Only for TCBOX<0-5> and I<0-5> counters)

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 8 bit hex value Set bits 24-31 in config register
match0 8 bit hex value Channel mask filter, sets bits 36-43 in config register Only for TCBOX<0-5> counters. Check 3rd Gen Intel® Xeon® Processor Scalable Family, Codename Ice Lake, Uncore Performance Monitoring Reference Manual for more information.
\anchor ICX_BBOX

Mesh2Memory counters

The Intel® Icelake SP microarchitecture provides measurements of the mesh (M2M) which connects the cores with the Uncore devices. The description from Intel®:
For all Boxes that must communicate with the Mesh, there are a common set of events to capture various kinds of information about traffic flowing through their connection to the Mesh. The same encodings are used to request the mesh events in each box.
The M2M devices is first introduced in the Intel® Skylake SP microarchitecture. There was no suitable unit name for this, so LIKWID calls them simply M2M.

Counter and events

Counter name Event name
M2M<0-3>C0 *
M2M<0-3>C1 *
M2M<0-3>C2 *
M2M<0-3>C3 *

Available Options

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 5 bit hex value Set bits 24-28 in config register
\anchor ICX_PBOX

Mesh2PCIe counters

The Intel® Icelake SP microarchitecture provides measurements of the mesh to the IIO devices. The description from Intel®:
M2PCIe blocks manage the interface between the Mesh and each IIO stack.

Counter and events

Counter name Event name
P<0-5>C0 *
P<0-5>C1 *
P<0-5>C2 *
P<0-5>C3 *

Available Options

Option Argument Operation Comment
edgedetect N Set bit 18 in config register
invert N Set bit 23 in config register
threshold 5 bit hex value Set bits 24-28 in config register