NOTE: This part of the manual is relevant only if you are using the full-system PTLsim/X. If you are looking for the userspace-only version, please skip this entire part and read Part II instead.
| 1]1WARNING: PTLsim/X assumes fairly high level of familiarity with Xen and the Linux kernel. If you have never compiled your own Linux kernel or if you are not yet running Xen or are unsure how to create and use domains, STOP NOW and become familiar with Xen itself before attempting to use PTLsim/X. The following sections all assume you are familiar with Xen, at least from a system administration perspective. We cannot provide support for Xen-related issues unless they are caused by PTLsim. |
Prerequisites:
All files listed below can be downloaded from https://ptlsim.org/download.php.
| 1]1IMPORTANT: The instructions below refer to specific versions of various files (i.e., Xen hypervisor, Linux kernel, etc.). We regularly update the versions of these files, and newer PTLsim/X versions may not work correctly with older kernel and/or hypervisor versions (i.e. the versions should be matching). The following instructions are therefore for informational purposes only; always check the PTLsim web site's download page for the latest versions of these files. The following versions are correct as of September 20th, 2007 |
PTLsim is run in domain 0 as root, for instance by using the ``sudo ptlsim ...'' command. The -domainN option is used to specify the domain to access. The following scenarios show by example how this is done.
In the following examples, we will assume the target domain is called ptlvm.
Start your domain as follows:
sudo xm list
sudo xm console domainname
The xm list command will print the domain ID assigned to ptlvm. On our test machine, the output looks like:
Using config file "ptlvm".
Started domain ptlvm
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 7877 4 r--- 137.9
ptlvm 21 128 1 -p-- 0.0
// PTLsim: Cycle Accurate x86-64 Full System Simulator
// Copyright 1999-2007 Matt T. Yourst <>
//
// Revision 225 (2007-09-21)
// Built Sep 21 2007 16:21:36 on tidalwave.lab.ptlsim.org using gcc-4.2
// Running on typhoon.lab.ptlsim.org
//
Processing -domain 21 -logfile ptlsim.log -native
System Information:
Running on hypervisor version xen-3.0-x86_64-ptlsim
Xen is mapped at virtual address 0xffff800000000000
PTLsim is running across 1 VCPUs:
VCPU 0: 2202 MHz
Memory Layout:
System: 524208 pages, 2096832 KB
Domain: 32768 pages, 131072 KB
PTLsim reserved: 8192 pages, 32768 KB
Page Tables: 275 pages, 1100 KB
PTLsim image: 407 pages, 1628 KB
Heap: 7510 pages, 30040 KB
Stack: 256 pages, 1024 KB
Interfaces:
PTLsim page table: 282898
Shared info mfn: 4056
Shadow shinfo mfn: 295164
PTLsim hostcall: event channel 3
PTLsim upcall: event channel 4
Switched to native mode
Linux version 2.6.18-mtyrel-k8-64bit-xen (yourst@tidalwave) (gcc version 4.1.0 (SUSE Linux)) #2 Sun Oct 8 02:29:10 EDT 2006
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 0000000008800000 (usable)
No mptable found.
Built 1 zonelists. Total pages: 34816
Kernel command line: nousb noide root=/dev/hda1 xencons=ttyS console=ttyS0
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 8192 bytes)
Xen reported: 2202.808 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Software IO TLB disabled
Memory: 123180k/139264k available (2783k kernel code, 7728k reserved, 959k data, 184k init)
Calibrating delay using timer specific routine.. 4407.14 BogoMIPS (lpj=2203570)
...
NET: Registered protocol family 1
NET: Registered protocol family 17
VFS: Mounted root (ext2 filesystem) readonly.
Welcome to the PTLsim demo machine!
root [ptlsim /] cat /proc/cpuinfo
At this point, we would like to start an actual simulation run. For purposes of illustration, this run is composed of three actions:
To conduct this simulation, the ptlctlcommand is used within the running virtual machine (by typing it at the domain's console); it is not run on the host system at all:
[1] 775
root [ptlsim /] ptlctl -core ooo -stopinsns 100m -run : -core seq -stopinsns 200m -run : -native
Sending flush and command list to PTLsim hypervisor:
-core ooo -stopinsns 100m -run
-core seq -stopinsns 200m -run
-native
PTLsim returned rc 0
root [ptlsim /]
The second ptlctl command submits the three simulation actions to PTLsim, separated by colons (``:'').
At the PTLsim console, the following output is produced (the cycle counters will update regularly):
Breakout request received from native mode
Switched to simulation mode
Returned from switch to native: now back in sim
Processing -core ooo -stopinsns 100m -run
Completed 75258330 cycles, 100000000 commits: 461819 cycles/sec, 795201, insns/sec
Processing -core seq -stopinsns 200m -run
Completed 200000000 cycles, 200000000 commits: 6941302 cycles/sec, 6941302, insns/sec
Processing -native
Switched to native mode
This powerful capability allows ``self-directed'' simulation scripts (i.e. standard shell scripts), in which ptlctlis run immediately before starting a benchmark program, then ptlctlis run again after the program exits to end the simulation and switch back to native mode.
In Section 10.3, the configuration options common to both userspace PTLsim and full system PTLsim/X wer listed. PTLsim/X also introduces a number of special options only applicable to full system simulation:
Actions:
Start a simulation run, using the core model specified by the -core option (the default core is ``ooo'').
Stop the simulation run currently in progress, and wait for further commands. This is generally issued from another console window.
Switch the domain to native mode.
Kill the domain. This is equivalent to ``xm destroy'', but it also allows PTLsim to perform cleanup actions and flush all files before exiting.
PTLsim/X provides the ability to send commands and modify configuration options in the running simulation from another console on the host system. This is different from how the ptlctl program is used inside the target domain to script simulations: in this case, the commands are submitted asynchronously from the host system.
For instance,
To reset the log level in the middle of a simulation run, use the following:
ptlsim: Sending request '-domain ptlvm -loglevel 99 : -run' to domain 12...OK
Most options (such as -loglevel, -stoprip, etc.) can be updated at any time in this manner.
To end a simulation currently in progress, use this:
PTLsim supports command scripts, in which a file containing a list of commands is passed on the PTLsim command line as follows:
-logfile ptlsim.log -loglevel 4 -stats ptlsim.stats -snapshot-cycles 10m
# Run the simulation
-core seq -run -stopinsns 20m
-core ooo -run -stopinsns 100m
-native # All done (switch to native mode)
Command scripts can be nested (i.e. a script can itself include other scripts using @scriptname). When multiple commands are given on the command line separated by colons (``:''), any @scriptname clauses are processed after all other commands on the command line.
| 1]1We maintain a tutorial on how to set up checkpoints and perform advanced checkpointing techniques at https://ptlsim.org/capswiki/index.php/SPEC_2006. Note that this address is subject to change. |
Xen provides the ability to capture the state of a domain into a checkpoint file stored on disk. PTLsim can leverage this capability to start simulation from a checkpoint, avoiding the need to go through the entire boot process, and allowing precisely reproducable results across multiple simulation runs.
To create a checkpoint, boot the domain in native mode without PTLsim running, and bring the domain to the point where you would like to begin simulation. Then, in another console, run:
To restore the domain to that checkpoint, run:
sudo xm list
sudo xm console ptlvm
To exit PTLsim, use ``sudo ptlsim -kill -domain X'' from another console. To abort PTLsim immediately, use Ctrl+C on the ptlsim process, then type ``xm kill ptlvm'' to destroy the domain.
Full system simulation poses some difficult philosophical questions about the nature of time itself and the relativistic phenomenon of ``time dilation''. Specifically, if a simulator runs X times slower than the native CPU, both external interrupts and timer interrupts should theoretically be generated X times slower than in the real world. This is critical for obtaining accurate simulation results: for events like network traffic, if a real network device fed interrupts into the domain in realtime, and the simulator injected these interrupts into the simulation at the same rate, they would appear to arrive thousands of times faster than any physical network interface could deliver them. This can easily result in a livelock situation not possible in a real machine; at the very least it will deliver misleading performance results.
On the other hand, interacting with a domain running at the ``correct'' rate according to its own simulated clock can be unpleasant for users. For instance, if the ``sleep 1'' command is run in a Linux domain under PTLsim, instead of sleeping for 1 second of wall clock time (as perceived by the user), the domain will wait until 1 billion cycles have been fully simulated (assuming the simulated processor frequency is 1 GHz). This is because PTLsim keys interrupt delivery and all timers to the simulated cycle number in which the interrupt should arrive (based on the core clock frequency). In addition to being annoying, this behavior will massively confuse network applications that rely on precise timing information: a TCP/IP endpoint outside the domain will not expect packets to arrive thousands of times slower than its own realtime clock expects, resulting in retransmissions and timeouts that would never occur if both endpoints were inside the same ``time dilated'' domain.
Rather than attempt to solve this philosophical dilemma, PTLsim allows users to choose the options that best suit their simulation accuracy needs. The following options control the notion of time inside the simulation:
Specify the CPU core frequency (in Hz) reported to the domain. To specify a 2.4 GHz core, use ``-corefreq 2400m''. This option is used to calculate the number of cycles between timer interrupts, as described below.
NOTE: If you plan on switching the domain between simulation and native mode, we strongly recommend avoiding this option, to allow the host machine frequency to match the simulated frequency.
Specify the timer interrupt frequency in interrupts per second. By default, 100 interrupts per second are used, since this is the standard for Linux kernels.
Hint: if keyboard interaction with the domain seems slow or sluggish, this is because Linux only flushes console buffers to the screen at every clock tick. Specifying -timerfreq 1000 will greatly improve interactive response at the expense of more interrupt overhead.
By default, the realtime clock reported to the domain is the current time of day. This option forces the clock to reset to whatever time the domain's checkpoint (if any) was created. This may allow better cycle accurate reproducibility of random number generators, for instance.
PTLsim normally delivers all interrupts at the time dilated rate, as described above. While this provides the most realistic simulation accuracy, it may be undesirable for some applications, particularly in networking. The -realtime option delivers external interrupts to the domain as soon as they arrive at PTLsim's interrupt handler; they are not deferred. The realtime clock reported to the domain is also not dilated; it is locked to the current wall clock time. This option does not affect the timer interrupt frequency; use the -timerfreq option to directly manipulate this.
Do not allow any external interrupts or events to reach the domain; only the timer interrupt is delivered at the specified rate by PTLsim. This mode is necessary to provide guaranteed reproducable cycle accurate behavior across runs; it eliminates almost all non-deterministic events (like outside device interrupts) from the simulation. However, it is not very practical, since disk and network access is impossible in this mode (since the Xen disk and network drivers could never wake up the domain when data arrives). This mode is most useful for debugging starting at a checkpoint, or when using a ramdisk with pre-scripted boot actions.
PTLsim/X has a few additional options related to full system simulation:
Reserves M megabytes of physical memory for PTLsim and its translation cache. The default is 32 MB; the valid range is from 16 MB to 512 MB. See Chapter 14 for details.