1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
|
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE article PUBLIC "-//FreeBSD//DTD DocBook XML V4.5-Based Extension//EN"
"../../../share/xml/freebsd45.dtd" [
<!ENTITY art.re.pkgs '<ulink url="&url.articles.releng-packages;/article.html">The Release Engineering of Third Party Packages</ulink>'>
]>
<article lang='en'>
<title>&os; Release Engineering</title>
<articleinfo>
<!-- This paper was presented at BSDCon Europe in Brighton, UK on
November 11, 2001 -->
<confgroup>
<confdates>November 2001</confdates>
<conftitle>BSDCon Europe</conftitle>
</confgroup>
<authorgroup>
<author>
<firstname>Murray</firstname>
<surname>Stokely</surname>
<authorblurb>
<para>I've been involved in the development of &os; based products
since 1997 at Walnut Creek CDROM, BSDi, and now Wind River Systems.
&os; 4.4 was the first official release of &os; that I played
a significant part in.</para>
</authorblurb>
<affiliation>
<address><email>murray@FreeBSD.org</email>
<otheraddr><ulink url="http://www.FreeBSD.org/~murray/"></ulink></otheraddr>
</address>
</affiliation>
</author>
</authorgroup>
<legalnotice id="trademarks" role="trademarks">
&tm-attrib.freebsd;
&tm-attrib.cvsup;
&tm-attrib.intel;
&tm-attrib.general;
</legalnotice>
<pubdate>$FreeBSD$</pubdate>
<releaseinfo>$FreeBSD$</releaseinfo>
<abstract>
<para>This paper describes the approach used by the &os;
release engineering team to make production quality releases
of the &os; Operating System. It details the methodology
used for the official &os; releases and describes the tools
available for those interested in producing customized &os;
releases for corporate rollouts or commercial
productization.</para>
</abstract>
</articleinfo>
<!-- Introduction -->
<sect1 id="introduction">
<title>Introduction</title>
<para>The development of &os; is a very open process. &os; is
comprised of contributions from thousands of people around the
world. The &os; Project provides anonymous
<acronym>CVS</acronym>[1] access to the general public so that
others can have access to log messages, diffs (patches) between
development branches, and other productivity enhancements that
formal source code management provides. This has been a huge help
in attracting more talented developers to &os;. However, I
think everyone would agree that chaos would soon manifest if write
access was opened up to everyone on the Internet. Therefore only
a <quote>select</quote> group of nearly 300 people are given write
access to the <acronym>CVS</acronym> repository. These
<emphasis>committers[5]</emphasis> are responsible for the bulk of
&os; development. An elected <emphasis>core-team[6]</emphasis>
of very senior developers provides some level of direction over
the project.</para>
<para>The rapid pace of <systemitem
class="osname">&os;</systemitem> development leaves little time
for polishing the development system into a production quality
release. To solve this dilemma, development continues on two
parallel tracks. The main development branch is the
<emphasis>HEAD</emphasis> or <emphasis>trunk</emphasis> of our CVS
tree, known as <quote>&os;-CURRENT</quote> or
<quote>-CURRENT</quote> for short.</para>
<para>A more stable branch is maintained, known as
<quote>&os;-STABLE</quote> or <quote>-STABLE</quote> for short.
Both branches live in a master CVS repository in California and
are replicated via <application
class="software">CVSup</application>[2] to mirrors all over the
world. &os;-CURRENT[7] is the <quote>bleeding-edge</quote> of
&os; development where all new changes first enter the system.
&os;-STABLE is the development branch from which major releases
are made. Changes go into this branch at a different pace, and
with the general assumption that they have first gone into
&os;-CURRENT and have been thoroughly tested by our user
community.</para>
<para>In the interim period between releases, monthly snapshots are
built automatically by the &os; Project build machines and made
available for download from <systemitem
class="resource">ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/</systemitem>.
The widespread availability of binary release snapshots, and the
tendency of our user community to keep up with -STABLE development
with CVSup and <quote><command>make</command>
<maketarget>world</maketarget></quote>[7] helps to keep
&os;-STABLE in a very reliable condition even before the
quality assurance activities ramp up pending a major
release.</para>
<para>Bug reports and feature requests are continuously submitted by
users throughout the release cycle. Problems reports are entered into our
<application class="software">GNATS</application>[8] database
through email, the &man.send-pr.1; application, or via the web
interface provided at <ulink
url="http://www.FreeBSD.org/send-pr.html"></ulink>.</para>
<para>To service our most conservative users, individual release
branches were introduced with &os; 4.3.
These release branches are created shortly before a final release
is made. After the release goes out, only the most critical
security fixes and additions are merged onto the release branch.
In addition to source updates via CVS, binary patchkits are
available to keep systems on the
<emphasis>RELENG_<replaceable>X</replaceable>_<replaceable>Y</replaceable></emphasis>
branches updated.</para>
<sect2>
<title>What this article describes</title>
<para>The following sections of this article describe:</para>
<variablelist>
<varlistentry>
<term><xref linkend="release-proc"/></term>
<listitem>
<para>The different phases of the release engineering process
leading up to the actual system build.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="release-build"/></term>
<listitem>
<para>The actual build process.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="extensibility"/></term>
<listitem>
<para>How the base release may be extended by third parties.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="lessons-learned"/></term>
<listitem>
<para>Some of the lessons learned through the release of &os; 4.4.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><xref linkend="future"/></term>
<listitem>
<para>Future directions of development.</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
<!-- Release Process -->
<sect1 id="release-proc">
<title>Release Process</title>
<para>New releases of &os; are released from the -STABLE branch
at approximately four month intervals. The &os; release
process begins to ramp up 45 days before the anticipated release
date when the release engineer sends an email to the development
mailing lists to remind developers that they only have 15 days to
integrate new changes before the code freeze. During this time,
many developers perform what have become known as <quote>MFC
sweeps</quote>. <acronym>MFC</acronym> stands for <quote>Merge
From CURRENT</quote> and it describes the process of merging a
tested change from our -CURRENT development branch to our -STABLE
branch.</para>
<sect2>
<title>Code Review</title>
<para>Thirty days before the anticipated release, the source
repository enters a <quote>code slush</quote>. During this
time, all commits to the -STABLE branch must be approved by the
&a.re;. The kinds of changes that are allowed during this 15 day
period include:</para>
<itemizedlist>
<listitem>
<para>Bug fixes.</para>
</listitem>
<listitem>
<para>Documentation updates.</para>
</listitem>
<listitem>
<para>Security-related fixes of any kind.</para>
</listitem>
<listitem>
<para>Minor changes to device drivers, such as adding new Device
IDs.</para>
</listitem>
<listitem>
<para>Any additional change that the release engineering team feels
is justified, given the potential risk.</para>
</listitem>
</itemizedlist>
<para>After the first 15 days of the code slush, a
<emphasis>release candidate</emphasis> is released for
widespread testing and the code enters a <quote>code
freeze</quote> where it becomes much harder to justify new
changes to the system unless a serious bug-fix or security issue
is involved. During the code freeze, at least one release
candidate is released per week, until the final release is
ready. During the days leading to the final release, the
release engineering team is in constant communication with the
security-officer team, the documentation maintainers, and the
port maintainers, to ensure that all of the
different components required for a successful release are
available.</para>
</sect2>
<sect2>
<title>Final Release Checklist</title>
<para>When several release candidates have been made available for
widespread testing and all major issues have been resolved, the
final release <quote>polishing</quote> can begin.</para>
<sect3 id="rel-branch">
<title>Creating the Release Branch</title>
<para>As described in the introduction, the
<literal>RELENG_<replaceable>X</replaceable>_<replaceable>Y</replaceable></literal>
release branch is a relatively new addition to our release
engineering
methodology. The first step in creating this branch is to
ensure that you are working with the newest version of the
<literal>RELENG_<replaceable>X</replaceable></literal> sources
that you want to branch <emphasis>from</emphasis>.</para>
<screen>/usr/src&prompt.root; <userinput>cvs update -rRELENG_4 -P -d</userinput></screen>
<para>The next step is to create a branch point
<emphasis>tag</emphasis>, so that diffs against the start of
the branch are easier with CVS:</para>
<screen>/usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4 RELENG_4_8_BP src</userinput></screen>
<para>And then a new branch tag is created with:</para>
<screen>/usr/src&prompt.root; <userinput>cvs rtag -b -rRELENG_4_8_BP RELENG_4_8 src</userinput></screen>
<note>
<para><emphasis>The
<literal>RELENG_<replaceable>*</replaceable></literal> tags
are restricted for use by the CVS-meisters and release
engineers.</emphasis></para>
</note>
<sidebar>
<para>A <quote><emphasis>tag</emphasis></quote> is CVS
vernacular for a label that identifies the source at a specific point
in time. By tagging the tree, we ensure that future release builders
will always be able to use the same source we used to create the
official &os; Project releases.</para>
</sidebar>
<mediaobject>
<imageobject>
<imagedata fileref="branches-head" align="center"/>
</imageobject>
<textobject>
<phrase>&os; Development Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng3" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 3.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng4" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 4.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng5" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 5.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng6" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 6.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng7" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 7.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng8" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 8.x STABLE Branch</phrase>
</textobject>
</mediaobject>
<mediaobject>
<imageobject>
<imagedata fileref="branches-releng9" align="center"/>
</imageobject>
<textobject>
<phrase>&os; 9.x STABLE Branch</phrase>
</textobject>
</mediaobject>
</sect3>
<sect3 id="versionbump">
<title>Bumping up the Version Number</title>
<para>Before the final release can be tagged, built, and
released, the following files need to be modified to reflect
the correct version of &os;:</para>
<itemizedlist>
<listitem>
<para><filename>doc/en_US.ISO8859-1/books/handbook/mirrors/chapter.xml
</filename></para>
</listitem>
<listitem>
<para><filename>doc/en_US.ISO8859-1/books/porters-handbook/book.xml
</filename></para>
</listitem>
<listitem>
<para><filename>doc/share/xml/freebsd.ent</filename></para>
</listitem>
<listitem>
<para><filename>src/Makefile.inc1</filename></para>
</listitem>
<listitem>
<para><filename>src/UPDATING</filename></para>
</listitem>
<listitem>
<para><filename>src/gnu/usr.bin/groff/tmac/mdoc.local</filename></para>
</listitem>
<listitem>
<para><filename>src/release/Makefile</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/share/xml/release.dsl</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/share/examples/Makefile.relnotesng</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/share/xml/release.ent</filename></para>
</listitem>
<listitem>
<para><filename>src/share/examples/cvsup/standard-supfile</filename></para>
</listitem>
<listitem>
<para><filename>src/sys/conf/newvers.sh</filename></para>
</listitem>
<listitem>
<para><filename>src/sys/sys/param.h</filename></para>
</listitem>
<listitem>
<para><filename>src/usr.sbin/pkg_install/add/main.c</filename></para>
</listitem>
<listitem>
<para><filename>www/en/docs/man.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/cgi/ports.cgi</filename></para>
</listitem>
<listitem>
<para><filename>ports/Tools/scripts/release/config</filename></para>
</listitem>
</itemizedlist>
<para>The release notes and errata files also need to be adjusted for the
new release (on the release branch) and truncated appropriately
(on the stable/current branch):</para>
<itemizedlist>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/relnotes/common/new.xml
</filename></para>
</listitem>
<listitem>
<para><filename>src/release/doc/en_US.ISO8859-1/errata/article.xml
</filename></para>
</listitem>
</itemizedlist>
<para><application>Sysinstall</application> should be updated to note
the number of available ports and the amount of disk space required
for the Ports Collection[4]. This information is currently kept in
<filename>src/usr.sbin/sysinstall/dist.c</filename>.</para>
<para>After the release has been built, a number of file should
be updated to announce the release to the world.</para>
<itemizedlist>
<listitem>
<para><filename>doc/share/images/articles/releng/branches-releng<replaceable>X</replaceable>.pic</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/advisories.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/includes.release.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/share/xml/includes.release.xsl</filename></para>
</listitem>
<listitem>
<para><filename>www/en/releases/*</filename></para>
</listitem>
<listitem>
<para><filename>www/en/releng/index.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/news/news.xml</filename></para>
</listitem>
<listitem>
<para><filename>www/en/search/web.atoz</filename></para>
</listitem>
<listitem>
<para><filename>src/share/misc/bsd-family-tree</filename></para>
</listitem>
</itemizedlist>
</sect3>
<sect3 id="versionbump-major">
<title>Preparing a new major release branch
(RELENG_<replaceable>X</replaceable>)</title>
<para>When a new major release branch, such as
<literal>RELENG_6</literal> is branched from HEAD, some
additional files must be updated before releases can be made
from this new branch.</para>
<itemizedlist>
<listitem>
<para><filename>src/share/examples/cvsup/stable-supfile</filename>
- must be updated to point to the new -STABLE branch, when
applicable.</para>
</listitem>
</itemizedlist>
</sect3>
<sect3>
<title>Creating Release Tags</title>
<para>When the final release is ready, the following command
will create the <literal>RELENG_4_8_0_RELEASE</literal>
tag.</para>
<screen>/usr/src&prompt.root; <userinput>cvs rtag -rRELENG_4_8 RELENG_4_8_0_RELEASE src</userinput></screen>
<para>The Documentation and Ports managers are responsible for
tagging the respective trees with the <literal>RELEASE_4_8_0</literal>
tag.</para>
<para>Occasionally, a last minute fix may be required
<emphasis>after</emphasis> the final tags have been created.
In practice this is not a problem, since <acronym>CVS</acronym>
allows tags to be manipulated with <command>cvs
tag -d <replaceable>tagname filename</replaceable></command>.
It is very important that any last minute changes be tagged
appropriately as part of the release. &os; releases must
always be reproducible. Local hacks in the release
engineer's environment are not acceptable.</para>
</sect3>
</sect2>
</sect1>
<!-- Release Building -->
<sect1 id="release-build">
<title>Release Building</title>
<para>&os; <quote>releases</quote> can be built by anyone with a
fast machine and access to a source repository. (That should be
everyone, since we offer anonymous CVS! See The Handbook for
details.) The <emphasis>only</emphasis> special requirement is
that the &man.md.4; device must be available. If the
device is not loaded into your kernel, then the kernel module
should be automatically loaded when &man.mdconfig.8; is executed
during the boot media creation phase. All of the tools necessary
to build a release are available from the CVS repository in
<filename>src/release</filename>. These tools aim to provide a
consistent way to build &os; releases. A complete release can
actually be built with only a single command, including the
creation of <acronym>ISO</acronym> images suitable for burning to
CDROM, installation floppies, and an FTP install directory. This
command is aptly named <command>make
release</command>.</para>
<sect2>
<title><command>make release</command></title>
<para>To successfully build a release, you must first populate
<filename>/usr/obj</filename> by running <command>make
world</command> or simply
<command>make
buildworld</command>. The release
target requires several variables be set properly to build a
release:</para>
<itemizedlist>
<listitem>
<para><makevar>CHROOTDIR</makevar> - The directory to be used as the
chroot environment for the entire release build.</para>
</listitem>
<listitem>
<para><makevar>BUILDNAME</makevar> - The name of the release to be
built.</para>
</listitem>
<listitem>
<para><makevar>CVSROOT</makevar> - The location of a CVS Repository.
</para>
</listitem>
<listitem>
<para><makevar>RELEASETAG</makevar> - The CVS tag corresponding to the
release you would like to build.</para>
</listitem>
</itemizedlist>
<para>If you do not already have access to a local CVS
repository, then you may mirror one with <ulink
url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/synching.html#CVSUP">CVSup</ulink>.
The supplied supfile,
<filename>/usr/share/examples/cvsup/cvs-supfile</filename>, is
a useful starting point for mirroring the CVS
repository.</para>
<para>If <makevar>RELEASETAG</makevar> is omitted, then the
release will be built from the <literal>HEAD</literal> (aka -CURRENT) branch.
Releases built from this branch are normally referred to as
<quote>-CURRENT snapshots</quote>.</para>
<para>There are many other variables available to customize the
release build. Most of these variables are documented at the
top of <filename>src/release/Makefile</filename>. The exact
command used to build the official &os; 4.7 (x86) release
was:</para>
<screen><command>make <literal>release CHROOTDIR=/local3/release \
BUILDNAME=4.7-RELEASE \
CVSROOT=/host/cvs/usr/home/ncvs \
RELEASETAG=RELENG_4_7_0_RELEASE</literal>
</command>
</screen>
<para>The release <filename>Makefile</filename> can be broken down into several distinct
steps.</para>
<itemizedlist>
<listitem>
<para>Creation of a sanitized system environment in a separate
directory hierarchy with <quote><command>make
<literal>installworld</literal></command></quote>.
</para>
</listitem>
<listitem>
<para>Checkout from CVS of a clean version of the system source,
documentation, and ports into the release build hierarchy.</para>
</listitem>
<listitem>
<para>Population of <filename>/etc</filename> and
<filename>/dev</filename> in the chrooted
environment.</para>
</listitem>
<listitem>
<para>chroot into the release build hierarchy, to make it harder for
the outside environment to taint this build.</para>
</listitem>
<listitem>
<para><command>make world</command>
in the chrooted environment.</para>
</listitem>
<listitem>
<para>Build of Kerberos-related binaries.</para>
</listitem>
<listitem>
<para>Build <filename>GENERIC</filename> kernel.</para>
</listitem>
<listitem>
<para>Creation of a staging directory tree where the binary
distributions will be built and packaged.</para>
</listitem>
<listitem>
<para>Build and installation of the documentation toolchain needed to
convert the documentation source (SGML) into HTML and text documents
that will accompany the release.</para>
</listitem>
<listitem>
<para>Build and installation of the actual documentation
(user manuals, tutorials, release notes, hardware compatibility lists,
and so on.)</para>
</listitem>
<listitem>
<para>Build of the <quote>crunched</quote> binaries used for
installation floppies.</para>
</listitem>
<listitem>
<para>Package up distribution tarballs of the binaries and sources.
</para>
</listitem>
<listitem>
<para>Create the boot media and a <quote>fixit</quote> floppy.</para>
</listitem>
<listitem>
<para>Create FTP installation hierarchy.</para>
</listitem>
<listitem>
<para><emphasis>(optionally)</emphasis> Create ISO images for
CDROM/DVD media.</para>
</listitem>
</itemizedlist>
<para>For more information about the release build infrastructure,
please see &man.release.7;.</para>
<note><para>It is important to remove any site-specific settings
from <filename>/etc/make.conf</filename>. For example, it would
be unwise to distribute binaries that were built on a system
with <varname>CPUTYPE</varname> set to a specific
processor.</para></note>
</sect2>
<sect2>
<title>Contributed Software (<quote>ports</quote>)</title>
<para>The <ulink url="http://www.FreeBSD.org/ports">&os; Ports
collection</ulink> is a collection of over &os.numports;
third-party software packages available for &os;. The &a.portmgr;
is responsible for maintaining a consistent ports tree that can be used
to create the binary packages that accompany official &os;
releases.</para>
<para>The release engineering activities for our collection of
third-party packages is beyond the scope of this document. A
separate article, &art.re.pkgs;, covers this topic
in depth.</para>
</sect2>
<sect2>
<title>Release ISOs</title>
<para>Starting with &os; 4.4, the &os; Project decided to
release all four ISO images that were previously sold on the
<emphasis>BSDi/Wind River Systems/FreeBSD Mall</emphasis>
<quote>official</quote> CDROM distributions. Each of the four
discs must contain a <filename>README.TXT</filename> file that
explains the contents of the disc, a
<filename>CDROM.INF</filename> file that provides meta-data for
the disc so that &man.sysinstall.8; can validate and use the
contents, and a <filename>filename.txt</filename> file that
provides a manifest for the disc. This
<emphasis>manifest</emphasis> can be created with a simple
command:</para>
<screen>/stage/cdrom&prompt.root; <userinput>find . -type f | sed -e 's/^\.\///' | sort > filename.txt</userinput></screen>
<para>The specific requirements of each CD are outlined below.</para>
<sect3>
<title>Disc 1</title>
<para>The first disc is almost completely created by
<command>make
release</command>. The only changes
that should be made to the <filename>disc1</filename> directory are the addition of
a <filename>tools</filename> directory, and as many popular
third party software packages as will fit on the disc. The
<filename>tools</filename> directory contains software that allow users to create
installation floppies from other operating systems. This disc
should be made bootable so that users of modern PCs do not
need to create installation floppy disks.</para>
<para>If a custom kernel of &os; is to be included, then
&man.sysinstall.8; and &man.release.7; must be updated to
include installation instructions. The relevant code is contained
in <filename>src/release</filename> and <filename>src/usr.sbin/sysinstall</filename>.
Specifically, the file <filename>src/release/Makefile</filename>, and
<filename>dist.c</filename>, <filename>dist.h</filename>,
<filename>menus.c</filename>, <filename>install.c</filename>, and
<filename>Makefile</filename> will need to be updated under
<filename>src/usr.sbin/sysinstall</filename>. Optionally, you may choose
to update <filename>sysinstall.8</filename>.</para>
</sect3>
<sect3>
<title>Disc 2</title>
<para>The second disc is also largely created by <command>make
release</command>. This disc contains a <quote>live
filesystem</quote> that can be used from &man.sysinstall.8; to
troubleshoot a &os; installation. This disc should be
bootable and should also contain a compressed copy of the CVS
repository in the <filename>CVSROOT</filename> directory and
commercial software demos in the <filename>commerce</filename>
directory.</para>
</sect3>
<sect3>
<title>Discs 3 and 4</title>
<para>The remaining two discs contain additional software
packages for &os;. The packages should be clustered so that
a package and all of its <emphasis>dependencies</emphasis> are
included on the same disc. More information about the
creation of these discs is provided in the &art.re.pkgs;
article.</para>
</sect3>
<sect3>
<title>Multi-volume support</title>
<para><application>Sysinstall</application> supports multiple
volume package installations. This requires that each disc
have an <filename>INDEX</filename> file containing all of the
packages on all volumes of a set, along with an extra field
that indicates which volume that particular package is on.
Each volume in the set must also have the
<literal>CD_VOLUME</literal> variable set in the
<filename>cdrom.inf</filename> file so that sysinstall can
tell which volume is which. When a user attempts to install a
package that is not on the current disc, sysinstall will
prompt the user to insert the appropriate one.</para>
</sect3>
</sect2>
</sect1>
<!-- Distribution -->
<sect1 id="distribution">
<title>Distribution</title>
<sect2 id="dist-ftp">
<title>FTP Sites</title>
<para>When the release has been thoroughly tested and packaged for
distribution, the master FTP site must be updated. The official
&os; public FTP sites are all mirrors of a master server that
is open only to other FTP sites. This site is known as
<hostid>ftp-master</hostid>. When the release is ready, the
following files must be modified on <hostid>ftp-master</hostid>:</para>
<variablelist>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/</filename></term>
<listitem>
<para>The installable FTP directory as output from <command>make
release</command>.</para>
</listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release/</filename></term>
<listitem><para>The complete package build for this
release.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/tools</filename></term>
<listitem><para>A symlink to
<filename>../../../tools</filename>.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/<replaceable>X.Y</replaceable>-RELEASE/packages</filename></term>
<listitem><para>A symlink to
<filename>../../../ports/<replaceable>arch</replaceable>/packages-<replaceable>X.Y</replaceable>-release</filename>.</para></listitem>
</varlistentry>
<varlistentry>
<term><filename>/pub/FreeBSD/releases/<replaceable>arch</replaceable>/ISO-IMAGES/<replaceable>X.Y</replaceable>/<replaceable>X.Y</replaceable>-RELEASE-<replaceable>arch</replaceable>-*.iso</filename></term>
<listitem><para>The ISO images. The <quote>*</quote> is
<filename>disc1</filename>, <filename>disc2</filename>, etc.
Only if there is a <filename>disc1</filename> and there is an
alternative first installation CD (for example a
stripped-down install with no windowing system) there may
be a <filename>mini</filename> as well.</para>
</listitem>
</varlistentry>
</variablelist>
<para>For more information about the distribution mirror
architecture of the &os; FTP sites, please see the <ulink
url="&url.articles.hubs;/">Mirroring &os;</ulink> article.</para>
<para>It may take many hours to two days after updating
<hostid>ftp-master</hostid> before a majority of the Tier-1 FTP
sites have the new software depending on whether or not a package
set got loaded at the same time. It is imperative that the release
engineers coordinate with the &a.mirror-announce; before announcing the general
availability of new software on the FTP sites. Ideally
the release package set should be loaded at least four
days prior to release day. The release bits should be
loaded between 24 and 48 hours before the planned release
time with <quote>other</quote> file permissions turned off.
This will allow the mirror sites to download it but the
general public will not be able to download it from the mirror
sites. Mail should be sent to &a.mirror-announce; at the time
the release bits get posted saying the release has been staged
and giving the time that the mirror sites should begin allowing
access. Be sure to include a time zone with the
time, for example make it relative to GMT.</para>
</sect2>
<sect2 id="dist-cdrom">
<title>CD-ROM Replication</title>
<para>Coming soon: Tips for sending &os; ISOs to a replicator
and quality assurance measures to be taken.</para>
</sect2>
</sect1>
<!-- Extensibility -->
<sect1 id="extensibility">
<title>Extensibility</title>
<para>Although &os; forms a complete operating system, there is
nothing that forces you to use the system exactly as we have
packaged it up for distribution. We have tried to design the
system to be as extensible as possible so that it can serve as a
platform that other commercial products can be built on top
of. The only <quote>rule</quote> we have about this is that if you
are going to distribute &os; with non-trivial changes, we
encourage you to document your enhancements! The &os; community
can only help support users of the software we provide. We
certainly encourage innovation in the form of advanced
installation and administration tools, for example, but we cannot
be expected to answer questions about it.</para>
<sect2>
<title>Creating Customized Boot floppies</title>
<para>Many sites have complex requirements that may require
additional kernel modules or userland tools be added to the
installation floppies. The <quote>quick and dirty</quote> way
to accomplish this would be to modify the staging directory of
an existing <command>make release</command> build hierarchy:</para>
<itemizedlist>
<listitem>
<para>Apply patches or add additional files inside the chroot
release build directory.</para>
</listitem>
<listitem>
<para><command>rm
${CHROOTDIR}/usr/obj/usr/src/release/release.[59]</command></para>
</listitem>
<listitem>
<para>rebuild &man.sysinstall.8;, the kernel, or whatever
parts of the system your change affected.</para>
</listitem>
<listitem>
<para><command>chroot ${CHROOTDIR} ./mk floppies
</command></para>
</listitem>
</itemizedlist>
<para>New release floppies will be located in
<filename>${CHROOTDIR}/R/stage/floppies</filename>.</para>
<para>Alternatively, the
<filename>boot.flp</filename> make
target can be called, or the filesystem
creating script,
<filename>src/release/scripts/doFS.sh</filename>, may be invoked
directly.</para>
<para>Local patches may also be supplied to the release build by
defining the <makevar>LOCAL_PATCH</makevar> variable in <command>make
release</command>.
</para>
</sect2>
<sect2>
<title>Scripting <command>sysinstall</command></title>
<para>The &os; system installation and configuration tool,
&man.sysinstall.8;, can be scripted to provide automated installs
for large sites. This functionality can be used in conjunction
with &intel; PXE[12] to bootstrap systems from the network, or
via custom boot floppies with a sysinstall script. An example
sysinstall script is available in the CVS tree as
<filename>src/usr.sbin/sysinstall/install.cfg</filename>.</para>
</sect2>
</sect1>
<!-- Lessons Learned -->
<sect1 id="lessons-learned">
<title>Lessons Learned from &os; 4.4</title>
<para>The release engineering process for 4.4 formally began on
August 1st, 2001. After that date all commits to the
<literal>RELENG_4</literal> branch of &os; had to be explicitly
approved by the &a.re;. The first
release candidate for the x86 architecture was released on August
16, followed by 4 more release candidates leading up to the final
release on September 18th. The security officer was very involved
in the last week of the process as several security issues were
found in the earlier release candidates. A total of over
<emphasis>500</emphasis> emails were sent to the &a.re; in
little over a month.</para>
<para>Our user community has made it very clear that the security
and stability of a &os; release should not be sacrificed for
any self-imposed deadlines or target release dates. The &os;
Project has grown tremendously over its lifetime and the need for
standardized release engineering procedures has never been more
apparent. This will become even more important as &os; is
ported to new platforms.</para>
</sect1>
<!-- Future Directions -->
<sect1 id="future">
<title>Future Directions</title>
<para>It is imperative for our release engineering activities to
scale with our growing userbase. Along these lines we are working
very hard to document the procedures involved in producing &os;
releases.</para>
<itemizedlist>
<listitem>
<para><emphasis>Parallelism</emphasis> - Certain portions of the
release build are actually <quote>embarrassingly
parallel</quote>. Most of the tasks are very I/O intensive,
so having multiple high-speed disk drives is actually more important than
using multiple processors in speeding up the <command>make
release</command> process. If multiple disks are used for
different hierarchies in the &man.chroot.2;
environment, then the CVS checkout of the <filename>ports</filename> and <filename>doc</filename> trees
can be happening simultaneously as the <command>make
world</command> on another disk. Using a
<acronym>RAID</acronym> solution (hardware or software) can
significantly decrease the overall build time.</para>
</listitem>
<listitem>
<para><emphasis>Cross-building releases</emphasis> - Building
IA-64 or Alpha release on x86 hardware? <command>make
TARGET=ia64 release</command>.
</para>
</listitem>
<listitem>
<para><emphasis>Regression Testing</emphasis> - We need better
automated correctness testing for &os;.</para>
</listitem>
<listitem>
<para><emphasis>Installation Tools</emphasis> - Our installation
program has long since outlived its intended life span.
Several projects are under development to provide a more
advanced installation mechanism. The libh project was one
such project that aimed to provide an intelligent new package
framework and GUI installation program.</para>
</listitem>
</itemizedlist>
</sect1>
<!-- Acknowledgements -->
<sect1 id="ackno">
<title>Acknowledgements</title>
<para>I would like to thank Jordan Hubbard for giving me the
opportunity to take on some of the release engineering
responsibilities for &os; 4.4 and also for all of his work
throughout the years making &os; what it is today. Of course
the release would not have been possible without all of the
release-related work done by &a.asami;, &a.steve;, &a.bmah;, &a.nik;,
&a.obrien;, &a.kris;, &a.jhb; and the rest of the &os; development
community. I would also like to thank &a.rgrimes;, &a.phk;, and others
who worked on the release engineering tools in the very early days
of &os;. This article was influenced by release engineering
documents from the CSRG[13], the NetBSD Project[10], and John
Baldwin's proposed release engineering process notes[11].</para>
</sect1>
<!-- Reference / Biblio Section -->
<sect1 id="biblio">
<title>References</title>
<para>[1] CVS - Concurrent Versions System
<ulink url="http://www.cvshome.org"></ulink></para>
<para>[2] CVSup - The CVS-Optimized General Purpose Network File Distribution
System <ulink url="http://www.polstra.com/projects/freeware/CVSup"></ulink>
</para>
<para>[3] <ulink url="http://pointyhat.FreeBSD.org"></ulink></para>
<para>[4] &os; Ports Collection
<ulink url="http://www.FreeBSD.org/ports"></ulink></para>
<para>[5] &os; Committers <ulink
url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/contributors/staff-committers.html"></ulink>
</para>
<para>[6] &os; Core Team
<ulink url="&url.base;/administration.html#t-core"></ulink></para>
<para>[7] &os; Handbook
<ulink url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook"></ulink>
</para>
<para>[8] GNATS: The GNU Bug Tracking System
<ulink url="http://www.gnu.org/software/gnats"></ulink>
</para>
<para>[9] &os; PR Statistics
<ulink url="http://www.FreeBSD.org/prstats/index.html"></ulink></para>
<para>[10] NetBSD Developer Documentation: Release Engineering
<ulink url="http://www.NetBSD.org/developers/releng/index.html"></ulink>
</para>
<para>[11] John Baldwin's &os; Release Engineering Proposal
<ulink url="http://people.FreeBSD.org/~jhb/docs/releng.txt"></ulink>
</para>
<para>[12] PXE Jumpstart Guide
<ulink
url="http://www.FreeBSD.org/doc/en_US.ISO8859-1/articles/pxe/index.html"></ulink>
</para>
<para>[13] Marshall Kirk McKusick, Michael J. Karels, and Keith Bostic:
<ulink url="http://docs.FreeBSD.org/44doc/papers/releng.html">
<emphasis>The Release Engineering of 4.3BSD</emphasis></ulink>
</para>
</sect1>
</article>
|