iscsi Provider
The iscsi provider provides probes for tracing iSCSI target activity.
This is a kernel provider built into the COMSTAR iSCSI target port provider. The COMSTAR iSCSI target and the user-land iSCSI target (/usr/sbin/iscsitgtd) are mutually exclusive. Only one of the targets can be enabled at a time. The COMSTAR iSCSI target DTrace provider provides all the probes that are provided by the user-land iSCSI provider, so that any DTrace script written for the userland provider built into the iSCSI target daemon (iscsitgtd) will work with the COMSTAR iSCSI target port provider as well without any modification. Since this provider instruments the iSCSI target activity, DTrace commands and scripts must be run on the iSCSI target server to observe these probes.
Probes
| SCSI Event | Probes |
|---|---|
| SCSI command/response | iscsi:::scsi-commandiscsi:::scsi-response |
| Data out/in/request (rtt) | iscsi:::data-sendiscsi:::data-receiveiscsi:::data-request |
| Login and logout command/response | iscsi:::login-commandiscsi:::login-responseiscsi:::logout-commandiscsi:::logout-response |
| NOP out/in (pings) | iscsi:::nop-receiveiscsi:::nop-send |
| Text and task command/response | iscsi:::task-commandiscsi:::task-responseiscsi:::text-commandiscsi:::text-response |
| Asynchronous message from target | iscsi:::async-send |
| Buffer dispatch and completion (not available with the USDT provider) | iscsi:::xfer-startiscsi:::xfer-done |
Arguments
| Probes | Variable | Type | Description |
|---|---|---|---|
| * | args[0] | conninfo_t * | connection info |
| * | args[1] | iscsiinfo_t * | common iSCSI properties |
| scsi-command | args[2] | scsicmd_t * | SCSI command block (cdb) |
| xfer-startxfer-done | args[2] | xferinfo_t * | Buffer info |
Types
All COMSTAR iSCSI target probes have the first and second argument in common:
args[0] conninfo_t * connection information
conninfo_t
typedef struct conninfo {
string ci_local; /* local host IP address */
string ci_remote; /* remote host IP address */
string ci_protocol; /* protocol ("ipv4", "ipv6") */
} conninfo_t;
The conninfo_t structure is used by NFSv4 provider, Fibre Channel provider and is intended for use by all
application protocol providers as the first argument to indicate some basic information about the connection.
args[1] iscsiinfo_t * common iSCSI properties
iscsiinfo_t
typedef struct iscsiinfo {
string ii_target; /* target iqn */
string ii_initiator; /* initiator iqn */
string ii_isid; /* initiator session identifier */
string ii_tsih; /* target session identifying handle */
string ii_transport; /* transport type ("iser-ib", "sockets") */
uint64_t ii_lun; /* target logical unit number */
uint32_t ii_itt; /* initiator task tag */
uint32_t ii_ttt; /* target transfer tag */
uint32_t ii_cmdsn; /* command sequence number */
uint32_t ii_statsn; /* status sequence number */
uint32_t ii_datasn; /* data sequence number */
uint32_t ii_datalen; /* length of data payload */
uint32_t ii_flags; /* probe-specific flags */
} iscsiinfo_t;
The iscsiinfo_t structure is used to provide identifying information about the target and the initiator and
also some PDU level information such as lun, data length and sequence numbers.
The third argument is only used for the SCSI command probe or the data transfer probe
args[2] scsicmd_t * SCSI command block (cdb)
scsicmd_t
typedef struct scsicmd {
uint64_t ic_len; /* CDB length */
uint8_t *ic_cdb; /* CDB data */
} scsicmd_t;
The scsicmd_t structure is used by the SCSI command probe and it contains information about the SCSI command
blocks and is intended for use by all the application protocols that deal with SCSI data.
Although the transport layer is transparent to the user, the COMSTAR iSCSI target also supports iSCSI over Remote DMA (RDMA), also known as iSER. Since the data transfer phases are mapped to Remote DMA (RDMA) operations in iSER, the data-send, data-receive and data-request probes cannot be used with iSER. Instead the xfer-start and xfer-done probes can be used to trace the data transfer irrespective of the transport used. The data-receive, data-request and data-send probes can be used when a user wants to track the SCSI Data-IN and Data-OUT PDUs specifically.
args[2] xferinfo_t * data transfer information
xferinfo_t
typedef struct xferinfo {
uintptr_t xfer_laddr; /* local buffer address */
uint32_t xfer_loffset; /* offset within the local buffer */
uint32_t xfer_lkey; /* access control to local memory */
uintptr_t xfer_raddr; /* remote virtual address */
uint32_t xfer_roffset; /* offset from the remote address */
uint32_t xfer_rkey; /* access control to remote virtual address */
uint32_t xfer_len; /* transfer length */
uint32_t xfer_type; /* Read or Write */
} xferinfo_t;
The xferinfo_t structure is used by the xfer-start and the xfer-done probes and contain information about the
data transfer. When the transport type is iSER, the remote buffer information is given by the xfer_raddr,
xfer_rkey and xfer_roffset fields. It is set to 0 when the transport type is sockets.
Examples
One-liners
Frequency of iSCSI command types:
# dtrace -n 'iscsi*::: { @[probename] = count(); }'
Frequency of iSCSI client IP addresses:
# dtrace -n 'iscsi*::: { @[args[0]->ci_remote] = count(); }'
Payload bytes by iSCSI command type:
# dtrace -n 'iscsi*::: { @[probename] = sum(args[1]->ii_datalen); }'
Payload byte distribution by iSCSI command type:
# dtrace -n 'iscsi*::: { @[probename] = quantize(args[1]->ii_datalen); }'
iscsiwho.d
This is a simple script to produce a report of the remote IP addresses and a count of iSCSI events. This is intended to provide a quick summary of iSCSI activity when run on the iSCSI target server:
#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEGIN
{
printf("Tracing... Hit Ctrl-C to end.\n");
}
iscsi*:::
{
@events[args[0]->ci_remote, probename] = count();
}
dtrace:::END
{
printf(" %-26s %14s %8s\n", "REMOTE IP", "iSCSI EVENT", "COUNT");
printa(" %-26s %14s %@8d\n", @events);
}
This output shows the host and the number of iSCSI operations:
# ./iscsiwho.d Tracing... Hit Ctrl-C to end. ^C REMOTE IP iSCSI EVENT COUNT 192.168.1.109 nop-receive 1 192.168.1.109 nop-send 1 192.168.1.109 scsi-response 126 192.168.1.109 scsi-command 208 192.168.1.109 data-request 1207 192.168.1.109 data-receive 1207
The fields are:
| field | description |
|---|---|
| REMOTE IP | IP address of the client |
| iSCSI EVENT | iSCSI event type |
| COUNT | Number of events traced |
The example output shows normal traffic. For this simple script, these event names are not translated beyond their iSCSI provider probe names, and require some thought to comprehend (they are from the perspective of the iSCSI target server).
iscsixfer.d
Although the transport layer is transparent to the user, the COMSTAR iSCSI target also supports iSCSI over Remote DMA (RDMA), also known as iSER. An iSER initiator should be able to read and write data from an iSER target at high data rates with relatively low CPU utilization compared to iSCSI using TCP/IP. In order to see the transport layer in use, display the ii_transport field from the iscsiinfo_t structure.
Since the data transfer phases are mapped to Remote DMA (RDMA) operations in iSER, the data-send, data-receive and data-request probes cannot be used with iSER. Instead here is a simple script to print an aggregation of all the data transferred between two points using the xfer-start probe. This can be used for iSCSI using TCP/IP and iSCSI over Remote DMA.
The data-receive, data-request and data-send probes can be used when a user wants to track the SCSI Data-IN and Data-OUT PDUs specifically (e.g. if the PDUs are received out of order, one might want to trace the ii_ttt, ii_datasn, ii_statsn etc.). To just get a trace of IO activity, the xfer-start/xfer-done probes should suffice.
#!/usr/sbin/dtrace -s
#pragma D option quiet
iscsi:::xfer-start
{
@[args[0]->ci_remote, args[2]->xfer_type] = sum(args[2]->xfer_len);
}
END
{
printf("%26s %10s %8s\n", "REMOTE IP", "READ/WRITE", "BYTES");
printa("%26s %10s %15@d\n", @);
}
This output shows the transfer of bytes:
# ./iscsixfer.d Tracing... Hit Ctrl-C to end. ^C REMOTE IP READ/WRITE BYTES 192.168.1.109 write 464 192.168.1.109 read 1024
The fields are:
| field | description |
|---|---|
| REMOTE IP | IP address of the client |
| READ/WRITE | read or write |
| BYTES | Number of bytes transferred |
Now if a user is interested in just seeing the data move (read or write) as it happens, one could use this script
#!/usr/sbin/dtrace -s
#pragma D option quiet
BEGIN
{
printf(" %-26s %8s %10s\n", "REMOTE IP", "BYTES", "READ/WRITE");
}
iscsi:::xfer-start
{
printf("%26s %%8d %108s\n", args[0]->ci_remote, args[2]->xfer_len, args[2]->xfer_type);
}
An interpretation for some of these events are:
| iSCSI event | Interpretation |
|---|---|
| scsi-command | A SCSI command was issued, such as a read or a write. Use other scripts for a breakdown on the SCSI command type. |
| data-send | Data was sent from the iSCSI target server to the client; the client is performing a read. |
| data-receive | Data was received on the iSCSI target server from the client. The client is performing a write. |
Sign up or Log in to add a comment or watch this page.
