1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<link href="style.css" rel="stylesheet" type="text/css" />
<title>Testing LLDB</title>
</head>
<body>
<div class="www_title">
The <strong>LLDB</strong> Debugger
</div>
<div id="container">
<div id="content">
<!--#include virtual="sidebar.incl"-->
<div id="middle">
<div class="post">
<h1 class="postheader">Testing LLDB</h1>
<div class="postcontent">
<p>
The LLDB test suite consists of Python scripts located under the
<tt>test</tt> directory. Each script contains a number of test cases and is usually
accompanied by a C (C++, ObjC, etc.) source file. Each test first compiles the
source file and then uses LLDB to debug the resulting executable. The tests verify
both the LLDB command line interface and the scripting API.
</p>
</div>
<h1 class="postheader">Running tests</h1>
<div class="postcontent">
<h2>Running the full test suite</h2>
<p>
<strong>Windows Note</strong>: In the examples that follow, any invocations of <code>python</code>
should be replaced with <code>python_d</code>, the debug interpreter, when running the test
suite against a debug version of LLDB.
</p>
<p>
The easiest way to run the LLDB test suite is to use the <tt>check-lldb</tt> build
target. By default, the <tt>check-lldb</tt> target builds the test programs with
the same compiler that was used to build LLDB. To build the tests with a different
compiler, you can set the <strong>LLDB_TEST_COMPILER</strong> CMake variable. It is possible to
customize the architecture of the test binaries and compiler used by appending -A
and -C options respectively to the CMake variable <strong>LLDB_TEST_USER_ARGS</strong>. For
example, to test LLDB against 32-bit binaries
built with a custom version of clang, do:
</p>
<code>
<br />> cmake -DLLDB_TEST_ARGS="-A i386 -C /path/to/custom/clang" -G Ninja
<br />> ninja check-lldb
</code>
<p>Note that multiple -A and -C flags can be specified to <tt>LLDB_TEST_USER_ARGS</tt>.</p>
<p>Note that on NetBSD you must export <tt>LD_LIBRARY_PATH=$PWD/lib</tt> in your environment. This is due to lack of
the <tt>$ORIGIN</tt> linker feature.</p>
<h2>Running a specific test or set of tests</h2>
<p>
In addition to running all the LLDB test suites with the "check-lldb" CMake target above, it is possible to
run individual LLDB tests. For example, to run the test cases defined in TestInferiorCrashing.py, run:
</p>
<code>
<br />> cd $lldb/test
<br />> python dotest.py --executable <path-to-lldb> -p TestInferiorCrashing.py ../packages/Python/lldbsuite/test
</code>
<p>
If the test is not specified by name (e.g. if you leave the <code>-p</code> argument off), LLDB will run all tests in
that directory:
</p>
<code>
<br />> python dotest.py --executable <path-to-lldb> functionalities/data-formatter
</code>
<p>
Many more options that are available. To see a list of all of them, run:
</p>
<code>
> python dotest.py -h
</code>
<p>
The dotest.py script runs tests in parallel by default.
To disable the parallel test running feature, use the
<code>--no-multiprocess</code> flag. The number of
concurrent tests is controlled by
the <code>LLDB_TEST_THREADS</code> environment variable
or the <code>--threads</code> command line parameter.
The default value is the number of CPU cores on your
system.
</p>
<p>
The parallel test running feature will handle an
additional <code>--test-subdir SUBDIR</code> arg. When
specified, SUBDIR is relative to the root test directory
and will limit all parallel test running to that
subdirectory's tree of tests.
</p>
<p>
The parallel test runner will run all tests within a
given directory serially, but will run multiple
directories concurrently. Thus, as a test writer, we
provide serialized test run semantics within a
directory. Note child directories are considered
entirely separate, so two child directories could be
running in parallel with a parent directory.
</p>
<h2>Running the test-suite remotely</h2>
<p>
Running the test-suite remotely is similar to the process of running a local test
suite, but there are two things to have in mind:
</p>
<ul>
<li>
You must have the <code>lldb-server</code> running on the remote system, ready to
accept multiple connections. For more information on how to setup remote
debugging see the <a href="remote.html">Remote debugging</a> page.
</li>
<li>
You must tell the test-suite how to connect to the remote system. This is
achieved using the <code>--platform-name</code>, <code>--platform-url</code> and
<code>--platform-working-dir</code> parameters to <code>dotest.py</code>. These
parameters correspond to the <code>platform select</code> and <code>platform
connect</code> LLDB commands. You will usually also need to specify the compiler and
architecture for the remote system.
</li>
</ul>
<p>
Currently, running the remote test suite is supported only with
<code>dotest.py</code> (or <code>dosep.py</code> with a single thread), but we
expect this issue to be addressed in the near future.
</p>
</div>
<div class="postfooter"></div>
<h1 class="postheader">Debugging test failures</h1>
<div class="postcontent">
<h2>Non-Windows platforms</h2>
<p>
On non-Windows platforms, you can use the <code>-d</code> option to <code>dotest.py</code> which will cause the script to wait
for a while until a debugger is attached.
</p>
<h2>Windows</h2>
<p>
On Windows, it is strongly recommended to use <a href="https://github.com/Microsoft/PTVS/releases">Python Tools for Visual Studio</a>
for debugging test failures. It can seamlessly step between native and managed code, which is very helpful when you need to step
through the test itself, and then into the LLDB code that backs the operations the test is performing. A quick guide to getting
started with PTVS is as follows:
<ul>
<li>Install PTVS</li>
<li>
Create a Visual Studio Project for the Python code.
<ul>
<li>Go to File -> New -> Project -> Python -> From Existing Python Code.</li>
<li>Choose <code>llvm/tools/lldb</code> as the directory containing the Python code.</li>
<li>
When asked where to save the <code>.pyproj</code> file, choose the folder <code>llvm/tools/lldb/pyproj</code>.
This is a special folder that is ignored by the <code>.gitignore</code> file, since it is not checked in.
</li>
</ul>
</li>
<li>Set <code>test/dotest.py</code> as the startup file</li>
<li>
Make sure there is a Python Environment installed for your distribution. For example, if you installed Python to
<code>C:\Python35</code>, PTVS needs to know that this is the interpreter you want to use for running the test suite.
<ul>
<li>Go to Tools -> Options -> Python Tools -> Environment Options</li>
<li>Click Add Environment, and enter <code>Python 3.5 Debug</code> for the name. Fill out the values correctly.</li>
</ul>
</li>
<li>
Configure the project to use this debug interpreter.
<ul>
<li>Right click the Project node in Solution Explorer</li>
<li>In the <code>General</code> tab, Make sure <code>Python 3.5 Debug</code> is the selected Interpreter.</li>
<li>In <code>Debug/Search Paths</code>, enter the path to your <code>ninja/lib/site-packages</code> directory.</li>
<li>
In <code>Debug/Environment Variables</code>, enter<br/>
<code>VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\</code>
</li>
<li>
If you want to enabled mixed mode debugging, check <code>Enable native code debugging</code> (this slows down debugging,
so enable it only on an as-needed basis.)
</li>
</ul>
</li>
<li>
Set the command line for the test suite to run.
<ul>
<li>Right click the project in solution explorer and choose the <code>Debug</code> tab.</li>
<li>Enter the arguments to <code>dotest.py</code>. Note you must add <code>--no-multiprocess</code></li>
<li>
Example command options:
<code>
<br/># quiet mode
<br/>-q
<br />--arch=i686
<br /># Path to debug lldb.exe
<br />--executable D:/src/llvmbuild/ninja/bin/lldb.exe
<br /># Directory to store log files
<br />-s D:/src/llvmbuild/ninja/lldb-test-traces
<br />-u CXXFLAGS -u CFLAGS
<br /># If a test crashes, show JIT debugging dialog.
<br />--enable-crash-dialog
<br /># Path to release clang.exe
<br />-C d:\src\llvmbuild\ninja_release\bin\clang.exe
<br /># Path to the particular test you want to debug.
<br />-p TestPaths.py
<br /># Root of test tree
<br />D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test
<br /># Required in order to be able to debug the test.
<br />--no-multiprocess
</code>
</li>
<li>
As copy-pastable command line:<br/>
<code>
-q --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces
-u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
-p TestPaths.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --no-multiprocess
</code>
</li>
</ul>
</li>
</ul>
</p>
</div>
<div class="postfooter"></div>
</div>
</div>
</div>
</div>
</body>
</html>
|
