1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
|
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//FreeBSD//DTD XHTML 1.0 Transitional-Based Extension//EN"
"http://www.FreeBSD.org/XML/share/xml/xhtml10-freebsd.dtd" [
<!ENTITY title "FreeBSD.org intrusion announced November 17th 2012">
]>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>&title;</title>
<cvs:keyword xmlns:cvs="http://www.FreeBSD.org/XML/CVS">$FreeBSD$</cvs:keyword>
</head>
<body class="navinclude.about">
<table class="tblbasic">
<tbody>
<tr>
<td><h2 align="center"><a name="announce">Security Incident on
FreeBSD Infrastructure</a></h2>
<b>From:</b> FreeBSD Security Officer <security-officer@FreeBSD.org><br />
<b>To:</b> FreeBSD Security <FreeBSD-security@FreeBSD.org><br />
<b>Bcc:</b> freebsd-announce@freebsd.org, freebsd-security-notifications@FreeBSD.org<br />
<b>Reply-To:</b> secteam@FreeBSD.org<br />
<b>Subject:</b> Security Incident on FreeBSD Infrastructure<br />
<p>On Sunday 11th of November, an intrusion was detected on two
machines within the FreeBSD.org cluster. The affected machines
were taken offline for analysis. Additionally, a large portion
of the remaining infrastructure machines were also taken offline
as a precaution.</p>
<p>We have found no evidence of any modifications that would put
any end user at risk. However, we do urge all users to read the
report available at
<a href="/news/2012-compromise.html">http://www.freebsd.org/news/2012-compromise.html</a>
and decide on any required actions themselves. We will continue
to update that page as further information becomes known. We do
not currently believe users have been affected given current
forensic analysis, but we will provide updated information if
this changes.</p>
<p>As a result of this event, a number of operational security
changes are being made at the FreeBSD Project, in order to
further improve our resilience to potential attacks. We plan,
therefore, to more rapidly deprecate a number of legacy services,
such as cvsup distribution of FreeBSD source, in favour of our
more robust Subversion, freebsd-update, and portsnap models.</p>
<p>More information is available at
<a href="/news/2012-compromise.html">http://www.freebsd.org/news/2012-compromise.html</a></p>
<p>Saturday November 17th, 2012</p>
</td>
</tr>
</tbody>
</table>
<br />
<h2><a name="toc">Table of Contents</a></h2>
<ul>
<li><a href="#announce">Announcement</a></li>
<li><a href="#update20130430">Update: 30th April 2013</a></li>
<li><a href="#update20130323">Update: 23rd March 2013</a></li>
<li><a href="#update20130303">Update: 3rd March 2013</a></li>
<li><a href="#update20121229">Update: 29th December 2012</a></li>
<li><a href="#update20121127">Update: 27th November 2012</a></li>
<li><a href="#update20121122">Update: 22nd November 2012</a></li>
<li><a href="#update20121118">Update: 18th November 2012</a></li>
<li><a href="#details">Initial Details: 17th November 2012</a></li>
<li><a href="#impact">What is the Impact?</a></li>
<li><a href="#done">What has FreeBSD.org done about this?</a></li>
<li><a href="#recommend">Recommendations</a></li>
</ul>
<h1><a name="update20130430">Update: April 30th, 2013</a></h1>
<p>Port managers and cluster administrators have completed the
restoration of binary package building in the last few weeks.
This has brought us back the continuous updates for the old-style
binary packages on the 8.x and 9.x -STABLE branches. Note that,
as beneficial consequences, Release Candidate builds for the 8.4
release cycle can now include binary packages on the install
media, and the Project was able to add the missing binary packages
retroactively for 9.1-RELEASE on i386 and amd64 platforms.</p>
<p>Port managers are currently working on introducing new-style (as
known as <q>pkgng</q>) binary packages in the coming months,
please check the <a
href="http://lists.freebsd.org/mailman/listinfo/freebsd-ports-announce/">
&os; ports announcements list</a> for further gradual status
updates.</p>
<p>This is planned to be the last status update to this page. An
official announcement will be sent to the <a
href="http://lists.freebsd.org/mailman/listinfo/freebsd-announce/">
&os; announcements mailing list</a> with the further details
soon.</p>
<h1><a name="update20130323">Update: March 23rd, 2013</a></h1>
<p>Port managers have successfully restored some of the Project's
binary package building capacity. There are some issues left
still to resolve, e.g. how to publish the resulting package sets
in a secure manner or how to build packages seamlessly for 8.x and
9.x systems on a recent 10.x system that the head node
("pointyhat") is running, but we are very close to finish with the
preparations required for providing binary packages for the
upcoming 8.4 and further releases.</p>
<h1><a name="update20130302">Update: March 3rd, 2013</a></h1>
<p>Redports underwent a full security audit, and as a result could
be brought back on line. This took place on the 5th February, and
since then more backend hardware has been added to bring it back
up to full strength. On 11th February, sanity checks for ports
have been turned back on, reenabling generation and update of the
INDEX files used. The portsnap(8) service has been switched from
CVS to SVN on 25th February. The binary package building
infrastructure has undergone a major security review, and as a
result many changes have been made to the code. The review
completed on the 16th February and we are now in the process of
bringing it up on new hardware. At this point, we expect new
binary packages to be available in 2-4 weeks.</p>
<h1><a name="update20121229">Update: December 29th, 2012</a></h1>
<p>With the exception of systems relating to the building and testing
of packages, all FreeBSD.org infrastructure has now been brought
back online. A full audit of the third party package build
infrastructure code ("pointyhat") and package testing infrastructure
("redports") continues, and neither system will be brought back
online until audits are complete.</p>
<p>As a result, FreeBSD 9.1-RELEASE will be published with only
minimal i386 and amd64 (x86_64) precompiled package sets available,
and with no packages available for other architectures. This
package set will be available on the DVD image, and are sufficient
to install either the GNOME or the KDE desktop environment. For any
other uses, or for any packages not included on the CD, either
using the most recently available -stable package collection or
compiling ports from the ports tree are recommended. Packages
for 9.1-RELEASE will be made available at a later date. Instructions
for obtaining and updating the ports tree can be found in the
<a href="/doc/handbook/ports-using.html">
FreeBSD Handbook</a>.</p>
<h1><a name="update20121127">Update: November 27th, 2012</a></h1>
<p>Due to the legacy third-party package build controller head
nodes being offlined pending reinstall, we have been unable to
build new package sets over the last two weeks. As a result,
FreeBSD 9.1-RELEASE has been delayed as it was felt that we
should not ship the release without at least a minimal package
set available. We are now in a position where we are once
again able to build third-party packages for both of our
<a href="/doc/en_US.ISO8859-1/articles/committers-guide/archs.html">
Tier-1 architectures</a> (i386 and amd64), and are planning on
releasing it within the next few days with only a slightly
limited set of packages. Please note that historically we have
also provided packages on a best-effort basis for some of our
Tier-2 architectures such as sparc64, ia64 and powerpc. We are
not currently expecting to be in a position to build any Tier-2
packages before FreeBSD 9.1 ships, so initially no precompiled
packages will be available for these platforms. We
may be in a position to provide some packages for these
architectures shortly after the release.</p>
<p>A few reports covering this incident on external tech news
websites have confused details relating to how this incident
was discovered. Over the last few weeks, many of our primary
cluster servers have been either physically relocated and/or
replaced with new hardware as part of work planned several
months in advance. The discovery of this incident was
unrelated to this ongoing cluster maintenance. Several
service outages in the days surrounding the incident were
correctly attributed to ongoing cluster work, and were not
related in any way to the compromise. In parallel with the
physical upgrades and relocation of servers, we are also
reworking the network layout in order to provide better
functionality, security, resilience, and to reduce any impact
from incidents such as this. Due in a large part to the
progress already made here, we were able to have full
confidence in many systems and services so quickly after the
compromised hosts on the legacy network segment were
discovered.</p>
<h1><a name="update20121122">Update: November 22nd, 2012</a></h1>
<p>Although not mentioned in the original report,
<a href="/doc/handbook/ctm.html">CTM</a> (another mechanism for
retrieving FreeBSD source) uses the master trusted Subversion
repository as the source of its data. Additionally, verification of
CTM-sourced trees has been completed against the Subversion tree,
confirming that there are no differences between the two. Our
experimental Git repository has been similarly verified.</p>
<p>Work continues on rebuilding internal infrastructure and reinstating
services taken down during the incident. Web interfaces to the old
CVS repositories (CVSweb), and to GNATS (our bug-tracking database)
have been restored amongst other services, and other internal hosts
are being examined and rebuilt where necessary. A full audit of the
package building infrastructure is ongoing.</p>
<p>The FreeBSD Project is investing significant effort into looking
into both medium and long term infrastructure improvements to increase
security of the FreeBSD cluster.</p>
<h1><a name="update20121118">Update: November 18th, 2012</a></h1>
<p>Newer portsnap(8) snapshots are once again available. The
generation of these had been suspended as part of the infrastructure
lockdown, however all machines involved have either been audited or
reinstalled and so we are now confident that these can be made
available once more.</p>
<p>The Subversion to CVS exporter is now up and running again.
Updates made to the Subversion repository will once again appear in
repositories available via csup/CVSup. Please note that the use of
these exports are still deprecated, and users are urged to move to
one of the supported methods (for example, freebsd-update(8),
portsnap(8), or Subversion) in order to obtain updates. Note also
that we are still currently unable to guarantee the integrity of
past history within the CVS repository, but are confident in the
integrity of checkouts from the top-of-tree of each branch.</p>
<p>Please note that due to infrastructure changes, the first update
through either portsnap(8) or csup(1) is likely to show changes to
a large number of files. This is nothing to worry about.</p>
<p>As mentioned in the original announcement, a package set uploaded in
preparation for the upcoming FreeBSD 9.1-RELEASE could not be verified,
and so was removed. In order to allow system integrators and end
users to verify that packages they may have downloaded are not from
this set, we have provided files containing both
<a href="/news/2012-compromise/sha256.sums.20121118.txt">sha256</a> and
<a href="/news/2012-compromise/md5.sums.20121118.txt">md5</a> checksums
for all removed packages.</p>
<h1><a name="details">November 17th, 2012</a></h1>
<h2>Initial details</h2>
<p>On Sunday 11th November 2012, two machines within the FreeBSD.org
infrastructure were found to have been compromised. These machines
were head nodes for the legacy third-party package building
infrastructure. It is believed that the compromise may have occurred
as early as the 19th September 2012.</p>
<p>The compromise is believed to have occurred due to the leak of an
SSH key from a developer who legitimately had access to the machines
in question, and was not due to any vulnerability or code exploit
within FreeBSD.</p>
<p>To understand the impact of this compromise, you must first
understand that the FreeBSD operating system is divided into two
parts: the "base" maintained by the FreeBSD community, and a large
collection of third-party "packages" distributed by the Project.
The kernel, system libraries, compiler, core command-line tools
(e.g., SSH client), and daemons (e.g., sshd(8)) are all in the
"base". Most information in this advisory refers only to
third-party packages distributed by the Project.</p>
<p>No part of the base FreeBSD system has been put at risk. At no
point has the intruder modified any part of the FreeBSD base system
software in any way. However, the attacker had access sufficient
to potentially allow the compromise of third-party packages. No
evidence of this has been found during in-depth analysis, however
the FreeBSD Project is taking an extremely conservative view on this
and is working on the assumption that third-party packages generated
and distributed within a specific window could theoretically have
been modified.</p>
<h2><a name="impact">What is the Impact?</a></h2>
<p>If you are running a system that has had no third-party packages
installed or updated on it between the 19th September and 11th
November 2012, you have no reason to worry.</p>
<p>The Source, Ports and Documentation Subversion repositories have been
audited, and we are confident that no changes have been made to them.
Any users relying on them for updates have no reason to worry.</p>
<p>We have verified the state of FreeBSD packages and releases currently
available on ftp.FreeBSD.org. All package sets for existing versions
of FreeBSD and all available releases have been validated and we can
confirm that the currently available packages and releases have not
been modified in any way.</p>
<p>A package set for the upcoming FreeBSD 9.1-RELEASE had been uploaded
to the FTP distribution sites in preparation for 9.1-RELEASE. We are
unable to verify the integrity of this package set, and therefore it
has been removed and will be rebuilt. Please note that as these
packages were for a future release, the standard <q>pkg_add -r</q>
tools to install packages could not have downloaded these packages
unless they were requested explicitly.</p>
<p>We unfortunately cannot guarantee the integrity of any packages
available for installation between 19th September 2012 and 11th
November 2012, or of any ports compiled from trees obtained via any
means other than through svn.freebsd.org or one of its mirrors.
Although we have no evidence to suggest any tampering took place
and believe such interference is unlikely, we have to recommend you
consider reinstalling any machine from scratch, using trusted
sources.</p>
<p>We can confirm that the freebsd-update(8) binary upgrade mechanism is
unaffected, as it uses an entirely separate infrastructure. We have
also verified that the most recently-available portsnap(8) snapshot
matches the ports Subversion repository, and so can be fully trusted.
Please note that as a precaution, newer portsnap(8) snapshots are
currently not being generated.</p>
<h2><a name="done">What has FreeBSD.org done about this?</a></h2>
<p>As soon as the incident came to light, the FreeBSD Cluster
Administration team took the following actions:</p>
<ul>
<li>Power down the compromised machines.</li>
<li>Power down all machines on which the attacker may have had
access.</li>
<li>Audit the SVN and Perforce repositories to:
<ul>
<li>Verify that there had been no server intrusion.</li>
<li>Verify that no malicious commits had been made to the
repository.</li>
<li>Verify that the SVN repository exactly matched a known-clean
off-site copy.</li>
</ul>
</li>
<li>Verify that all FreeBSD base release media and install files on
the master FTP distribution sites are clean.</li>
<li>Verify all package sets available have checksums that match
known-good copies stored off-site.</li>
<li>The package set built for the upcoming 9.1-RELEASE did not have
an offsite backup to verify against. These have been deleted, and
will be rebuilt before 9.1 is released.</li>
<li>All suspect machines are being either reinstalled, retired, or
thoroughly audited before being brought back online.</li>
</ul>
<h2><a name="recommend">At this time, we recommend:</a></h2>
<ul>
<li>If you use the already-deprecated cvsup/csup distribution
mechanisms, you should stop now.</li>
<li>If you were using cvsup/csup for ports, you should switch to
portsnap(8) right away. Ports developers should be using
Subversion already. Further information on preferred mechanisms
for obtaining and updating the ports tree can be found at
<a href="/doc/handbook/ports-using.html">
http://www.freebsd.org/doc/handbook/ports-using.html</a></li>
<li>If you were using cvs/anoncvs/cvsup/csup for src, you should
consider either freebsd-update(8) for signed binary distribution
or Subversion for source. Please see the chapter on <a
href="/doc/handbook/updating-upgrading.html">updating
FreeBSD from source</a> in the handbook. Further details on
using Subversion and a list of official mirrors can be found
at <a href="/doc/handbook/svn.html">
http://www.freebsd.org/doc/handbook/svn.html</a></li>
<li>If you use portsnap(8), you should <tt>portsnap fetch &&
portsnap extract</tt> to the most recent snapshot. The most recent
portsnap(8) snapshot has been verified to exactly match the audited
Subversion repository. Please note that as a precaution, portsnap(8)
updates have been suspended temporarily.</li>
<li>Follow best practice security policies to determine how your
organization may be affected.</li>
<li>Conduct an audit of your system that uses FreeBSD.org provided
binary packages. Anything that may have been installed during the
affected period should be considered suspect. Although we have no
evidence of any tampering of any packages, you may wish to consider
rebuilding any affected machine from scratch, or if that is not
possible, rebuild your ports/packages.</li>
</ul>
<p>If you have any further questions about this announcement, please
contact the <a href="mailto:FreeBSD-security@FreeBSD.org">
FreeBSD-security@FreeBSD.org</a> mailing list, or for questions
where public mailing list distribution is inappropriate,
please contact the <a href="mailto:secteam@FreeBSD.org">FreeBSD
Security Team</a>.</p>
<p>This page will be updated as further information is known.</p>
</body>
</html>
|
