diff options
Diffstat (limited to 'en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv')
| -rw-r--r-- | en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv | 4300 |
1 files changed, 4300 insertions, 0 deletions
diff --git a/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv b/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv new file mode 100644 index 0000000000..16dcca9e84 --- /dev/null +++ b/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv @@ -0,0 +1,4300 @@ +0:00:09.469,0:00:11.309 +Hello my name is Marshall Kirk McKusick + +0:00:11.309,0:00:15.389 +and I've been around as long as dinosaurs +and mainframes have ruled the world + +0:00:15.389,0:00:18.429 +which is to say the sixties and seventies + +0:00:18.429,0:00:22.460 +however by 1970s a new breed of mammals had begun to show up +on the scene + +0:00:22.460,0:00:24.240 +known as mini computers + +0:00:24.240,0:00:28.230 +although they were just toys in the 1970s they would soon grow + +0:00:28.230,0:00:31.689 +and take over most of the computing market + +0:00:31.689,0:00:33.150 +In 1970 + +0:00:33.150,0:00:37.910 +at AT&T Bell laboratories two researchers Ken +Thompson and Dennis Ritchie began developing the + +0:00:37.910,0:00:39.900 +UNIX operating system + +0:00:39.900,0:00:42.040 +Ken Thompson who had been an alumnus at Berkeley + +0:00:42.040,0:00:46.100 +came back on a sabbatical in 1975 bringing UNIX +with him + +0:00:46.100,0:00:47.539 +In the year that he was there + +0:00:47.539,0:00:51.330 +he managed to get a number of graduate students interested +in UNIX + +0:00:51.330,0:00:53.940 +and by the time he left in 1976 + + +0:00:53.940,0:00:56.829 +Bill Joy has taken over in running the UNIX system + +0:00:56.829,0:01:00.470 +and in fact continuing to develop software for it. + +0:01:00.470,0:01:04.339 +Bill began packaging up the software that had +been developed under Berkeley UNIX and + +0:01:04.339,0:01:05.779 +and distributing it + +0:01:05.779,0:01:08.040 +as the Berkeley Software Distributions + +0:01:08.040,0:01:12.310 +whose name was quickly shortened to simply BSD + +0:01:12.310,0:01:16.330 +BSD continued to be distributed with +yearly distributions for almost fifteen + +0:01:16.330,0:01:17.490 +years + +0:01:17.490,0:01:21.920 +initially under Bill Joy and later under others including +yours truly. + +0:01:21.920,0:01:24.860 +By the late 1980s interest had began to grow + +0:01:24.860,0:01:27.400 +in freely redistributable software + +0:01:27.400,0:01:30.170 +so a number of us at Berkeley began separating +out + +0:01:30.170,0:01:32.649 +the AT&T proprietary bits of BSD + +0:01:32.649,0:01:35.710 +from those parts that were freely redistributable. + +0:01:35.710,0:01:40.590 +By the time of the final distribution at BSD +in 1992 + +0:01:40.590,0:01:43.620 +the entire distribution was freely redistributable. + +0:01:43.620,0:01:45.909 +I live in a capsule history here + +0:01:45.909,0:01:48.009 +but if you're interested in the entire story + +0:01:48.009,0:01:50.789 +I have this three-and-an-half hour epic + +0:01:50.789,0:01:54.590 +which is available from my website www.mckusick.com + +0:01:54.590,0:01:58.200 +that gives the entire history of Berkeley. + +0:01:58.200,0:02:00.239 +Following the final distribution from Berkeley + +0:02:00.239,0:02:01.450 +two groups sprung up + +0:02:01.450,0:02:03.600 +to continue supporting BSD + +0:02:03.600,0:02:08.080 +the first of this was the NetBSD whose primary +goal was to support + +0:02:08.080,0:02:10.459 +as many different architectures as possible + +0:02:10.459,0:02:14.769 +everything from your microwave oven all way +upto your cray XMP + +0:02:14.769,0:02:19.409 +In fact today NetBSD supports nearly +sixty architectures. + +0:02:19.409,0:02:22.419 +The other group that sprang up was FreeBSD. + +0:02:22.419,0:02:28.239 +Their goal was to bring up BSD and support +as wide a set of devices as possible on the + +0:02:28.239,0:02:29.719 +PC architecture. + +0:02:29.719,0:02:36.549 +They also had a goal of trying to make the + system as easy to install as possible to + +0:02:36.549,0:02:39.309 +attract by a wide group of developers + +0:02:39.309,0:02:42.319 +I chose to work primarily with the FreeBSD +group + +0:02:42.319,0:02:43.740 +both doing software + +0:02:43.740,0:02:46.140 +and also together with George Neville Neil + +0:02:46.140,0:02:51.069 +writing this book ""The Design and Implementation +of the FreeBSD Operating System"". + +0:02:51.069,0:02:52.060 +Together with this book + +0:02:52.060,0:02:53.959 +I developed a course + +0:02:53.959,0:02:56.500 +which runs for twelve chapters + +0:02:56.500,0:02:58.179 +and thirty hours. + +0:02:58.179,0:02:59.749 +The purpose of this video + +0:02:59.749,0:03:01.089 +is to give you a taste + +0:03:01.089,0:03:02.819 +of that course. + +0:03:02.819,0:03:07.249 +What follows are excerpts from the first lecture +of the course + +0:03:07.249,0:03:11.139 +which of course you can also get from my website +www.mckusick.com. + +0:03:11.139,0:03:13.069 + + +0:03:13.069,0:03:17.739 +Enjoy. + +0:03:17.739,0:03:22.239 +This class is nominally about FreeBSD +because well + +0:03:22.239,0:03:26.379 +that's what I know best and that's what +the textbook is organized around + +0:03:26.379,0:03:29.979 +but the fact of the matter is that it's really + +0:03:29.979,0:03:32.339 +a class about your UNIX and that + +0:03:32.339,0:03:36.539 +really covers sort of the broad range of things +in the open source arena as its FreeBSD + +0:03:36.539,0:03:37.689 +in Linux + +0:03:37.689,0:03:38.899 +which of course + +0:03:38.899,0:03:41.159 +you use a lot out + +0:03:41.159,0:03:41.550 +and + +0:03:41.550,0:03:44.349 +it also covers a commercial systems + +0:03:44.349,0:03:46.950 +%uh Solaris, HP-UX, + +0:03:46.950,0:03:49.279 +AIX and so on. + +0:03:49.279,0:03:52.419 +I am going to tend more towards the open +side + +0:03:52.419,0:03:56.389 +open source side of things.So it's really +going to be more FreeBSD in Linux than it's + +0:03:56.389,0:03:57.579 +going to be + +0:03:57.579,0:04:00.849 +Solaris and HP-UX and so on. + +0:04:00.849,0:04:06.959 +For the most part at the level of this course +we're dealing with the interfaces to the system + +0:04:06.959,0:04:07.329 +and + +0:04:07.329,0:04:11.599 +the fact that the matter is a those interfaces are highly +standardized at this point + +0:04:11.599,0:04:12.060 +and + +0:04:12.060,0:04:15.280 +whether it's FreeBSD or Linux or Solaris +or whatever + +0:04:15.280,0:04:19.460 +the Socket system call has to do the same +thing, it has to have the same arguments + +0:04:19.460,0:04:20.150 +in that, + +0:04:20.150,0:04:23.909 +it has to have the same effect + +0:04:23.909,0:04:27.319 +and so until you get down to the really nitty +details + +0:04:27.319,0:04:29.600 +of how they actually go about implementing +that + +0:04:29.600,0:04:31.960 +the differences are relatively minor. + +0:04:31.960,0:04:35.830 +So I would say that sixty to seventy percent +of the material that I'm covering + +0:04:35.830,0:04:40.779 +is just as true for FreeBSD as it would +be for Linux + +0:04:40.779,0:04:42.580 +or for Solaris + +0:04:42.580,0:04:44.659 +%uh AIX is a little bit + +0:04:44.659,0:04:45.629 +sort of off in the weeds + +0:04:45.629,0:04:48.709 +%uh as is HP-UX + +0:04:48.709,0:04:51.099 +but luckily we don't have to worry too much about +that. + +0:04:51.099,0:04:54.569 +Okay so + +0:04:54.569,0:04:59.279 +the other thing is that I'm going to assume that +all of you have used the system. I get + +0:04:59.279,0:05:00.910 +really sort of worried when people + +0:05:00.910,0:05:04.249 +you know raise the hands and ""Hey, what's a Shell?"" + +0:05:04.249,0:05:07.990 +or I don't +put a lot of code up but a one piece of code and someone said ""Why + +0:05:07.990,0:05:11.819 +are there two pipe symbols in the middle of +that that If statement?"". + +0:05:11.819,0:05:15.740 +No we're not programming the Shell we're programming +in C. + +0:05:15.740,0:05:19.970 +So hopefully you can tell the difference between +Shell scripts and C code. + +0:05:19.970,0:05:21.990 +so okay but I am but am gonna assume + +0:05:21.990,0:05:24.610 +you haven't really looked inside the system. + +0:05:24.610,0:05:28.289 +So I gonna start everything to at a very +high level. + +0:05:28.289,0:05:32.969 +The problem is I have already discovered you come +from a lot of different sort of + +0:05:32.969,0:05:33.819 +backgrounds + +0:05:33.819,0:05:35.180 +and + +0:05:35.180,0:05:36.280 +levels of knowledge + +0:05:36.280,0:05:37.900 +and so + +0:05:37.900,0:05:42.620 +the way that I find works best to sort of +be useful to everybody is that three pass + +0:05:42.620,0:05:43.860 +algorithm + +0:05:43.860,0:05:49.060 +so what I will do is start the first pass a very +broad brush high level + +0:05:49.060,0:05:50.569 +description of what's going on + +0:05:50.569,0:05:54.719 +and then I will go back and I'll go through the +same material again but at a lower level of + +0:05:54.719,0:05:55.300 +detail + +0:05:55.300,0:05:59.939 +then I finally go back and go through a very nittily +low-level of detail + +0:05:59.939,0:06:04.649 +and the fact of this is if you are learning new stuff +as I'm doing the high-level thing + +0:06:04.649,0:06:08.649 +you are gonna be utterly washed by the time I get to +low level niggly details + +0:06:08.649,0:06:10.699 +but since I'm going to do it topic by topic + +0:06:10.699,0:06:14.190 +when I get to the end of one of those nearly +low level niggly details + +0:06:14.190,0:06:17.900 +I'll give you a clue as I will say ""Brain +reset, I'm starting a new topic"" so even if + +0:06:17.900,0:06:19.330 +you're completely lost + +0:06:19.330,0:06:23.530 +you can now start listening again plus I'm gonna get +the broad brush up again. + +0:06:23.530,0:06:27.059 +okay and for those of you that know a lot of +this stuff already + +0:06:27.059,0:06:31.770 +you'll probably find the broad brush rather boring + +0:06:31.770,0:06:35.759 +but by the time we get down to nearly low level +details I think you'll actually + +0:06:35.759,0:06:37.860 +pick up some things that you will find + +0:06:37.860,0:06:39.710 +useful and interesting. + +0:06:39.710,0:06:43.759 +So in this way hopefully everybody will +get some + +0:06:43.759,0:06:47.699 +useful percentage of material out of the course. + +0:06:47.699,0:06:49.599 +I am gonna start out by just + +0:06:49.599,0:06:53.089 +walking through and giving you the + +0:06:53.089,0:06:56.919 +outline of what we're going to try and do here +here + +0:06:56.919,0:07:01.169 +As I said we're going to go roughly + +0:07:01.169,0:07:03.270 +just about two-and-an-half hours of lecture + +0:07:03.270,0:07:04.729 +about two hours forty minutes + +0:07:04.729,0:07:06.499 +per week + +0:07:06.499,0:07:07.619 +and + +0:07:07.619,0:07:11.770 +so we will start off this week with an introduction. + +0:07:11.770,0:07:13.860 +This is as I said we're going to start from the +top + +0:07:13.860,0:07:15.749 +and then just start working our way down + +0:07:15.749,0:07:19.350 +so the general thing I'm going to do is +to talk about the interface + +0:07:19.350,0:07:21.439 +%uh which is something that you + +0:07:21.439,0:07:25.319 +are presumably fairly familiar with since +you've worked with that system + +0:07:25.319,0:07:27.249 +and then + +0:07:27.249,0:07:29.739 +you have to sort of layout terminology + +0:07:29.739,0:07:32.080 +although we use normal English words + +0:07:32.080,0:07:34.419 +they have + +0:07:34.419,0:07:38.580 +sometimes rather bizarre meanings compared to their +common usage + +0:07:38.580,0:07:39.220 +and + +0:07:39.220,0:07:42.330 +so I will just sort of lay out the terminology +lay out the + +0:07:42.330,0:07:45.750 +the way we talk about how the system is structured + +0:07:45.750,0:07:50.780 +and this week we will also talk about the +basic services ""What is it that the kernel is + +0:07:50.780,0:07:52.929 +providing for us?"" + +0:07:52.929,0:07:54.060 +and then of course + +0:07:54.060,0:07:58.499 +we'll proceed to dive down in and and see how +that is done + +0:07:58.499,0:07:59.970 +so here in + +0:07:59.970,0:08:01.400 +Week number 2 + +0:08:01.400,0:08:05.450 +we're gonna look at the system from the +perspective of + +0:08:05.450,0:08:07.039 +something that + +0:08:07.039,0:08:08.720 +manages processes. + +0:08:08.720,0:08:12.170 +One way of looking at the kernel is it's really +just a + +0:08:12.170,0:08:16.440 +the resource manager and the resource that +its managing are things going to do with processes + +0:08:16.440,0:08:19.460 +So we'll look at a process, what the structure of +it is + +0:08:19.460,0:08:20.649 +and + +0:08:20.649,0:08:23.559 +talk about the different ways that they can +be structured. + +0:08:23.559,0:08:28.379 +Process can for example be an address space +and can have one thread running in it can have + +0:08:28.379,0:08:29.749 +multiple threads running in it. + +0:08:29.749,0:08:34.620 +so we'll talk about the different ways +that we think a process is. + +0:08:34.620,0:08:38.480 +We will look at the management of those processes + + +0:08:38.480,0:08:39.239 +we've got + +0:08:39.239,0:08:42.020 +to lay out the bits and pieces that +need to be managed + +0:08:42.020,0:08:44.660 +and then talk about + +0:08:44.660,0:08:47.190 +how we do that. + +0:08:47.190,0:08:51.740 +we'll talk about jails.. this is something +that you currently find only in FreeBSD + +0:08:51.740,0:08:55.060 +hasn't made it into + +0:08:55.060,0:08:56.320 +Linux yet although + +0:08:56.320,0:09:01.630 +the concept is being actively worked +on so my guess is that you'll see that + +0:09:01.630,0:09:03.500 +fairly soon. + +0:09:03.500,0:09:06.360 +we'll also then talk about scheduling + +0:09:06.360,0:09:10.579 +which is in essence how we decide what gets +to run, when it gets to run, how long it gets + +0:09:10.579,0:09:13.500 +to run, etc. + +0:09:13.500,0:09:14.330 +okay + +0:09:14.330,0:09:19.020 +The week after that we will go into virtual +memory. + +0:09:19.020,0:09:23.800 +Signals aren't really part of virtual memory +but they didn't fit into next week's + +0:09:23.800,0:09:26.400 +material so I just would dropped that at the +beginning + +0:09:26.400,0:09:29.850 +but the bulk of Week 3 is going to +be + +0:09:29.850,0:09:32.019 +the management of Virtual Memory. So we've got + +0:09:32.019,0:09:35.119 +a bunch of physical memory, a bunch of +processes that are + +0:09:35.119,0:09:37.940 +trying to use their address spaces + +0:09:37.940,0:09:39.590 +and we will talk about + +0:09:39.590,0:09:41.410 +essentially how you will make that all work + +0:09:41.410,0:09:43.510 +It's called a virtual memory because it's + +0:09:43.510,0:09:47.420 +sort of a cheat. We promise you the world and +then we deliver you + +0:09:47.420,0:09:51.480 +as small number of pages as we think we +can get away with. + +0:09:51.480,0:09:56.420 +Okay. So the first three weeks then essentially +get us through + +0:09:56.420,0:09:58.340 +looking at the world as if it was all + +0:09:58.340,0:10:00.560 +all about processes. + +0:10:00.560,0:10:03.880 +Then in Week 4 we change gears. we say +okay well you know + +0:10:03.880,0:10:07.570 +the kernel isn't just all about processes. You can sort of +look at it orthogonally and you can + +0:10:07.570,0:10:10.000 +say it's really just a giant I/O switch + +0:10:10.000,0:10:12.910 +it's just like a traffic cop that's just managing +these + +0:10:12.910,0:10:14.860 +I/O streams + +0:10:14.860,0:10:15.450 +and + +0:10:15.450,0:10:18.610 +so let's look at it from that perspective. + +0:10:18.610,0:10:19.310 +And + +0:10:19.310,0:10:24.740 +we'll start with special files, again this +sort of the interface when you talk about UNIX + +0:10:24.740,0:10:25.880 +systems, when you talk about + +0:10:25.880,0:10:27.950 +what's normally /dev + +0:10:27.950,0:10:34.170 +interface that gets you access +to the various I/O streams that are available + +0:10:34.170,0:10:37.220 +and we'll look at how that's organized and +the structure of it + +0:10:37.220,0:10:41.840 +which used to be fairly simple but in the +last decade has gotten + +0:10:41.840,0:10:43.670 +incredibly complicated. + +0:10:43.670,0:10:48.540 +We will also talk about pseudo terminals in +job control + +0:10:48.540,0:10:53.330 +this is about as interesting as watching the +grass grow but unfortunately it's + +0:10:53.330,0:10:55.490 +a major component of the system + +0:10:55.490,0:10:59.520 +and especially people that deal with system +administration have to know far more about + +0:10:59.520,0:11:06.520 +this than they probably ever thought they +wanted to. + +0:11:06.900,0:11:11.430 +Okay we will then continue in Week 5 with +the kernel I/O structure, + +0:11:11.430,0:11:16.090 +We will start with multiplexing of I/O. The +kernel of course has done this + +0:11:16.090,0:11:17.360 +always + +0:11:17.360,0:11:22.110 +but we're really talking more about how do +we export I/O multiplexing to + +0:11:22.110,0:11:25.970 +user applications. + +0:11:25.970,0:11:29.250 +We will then move into auto configuration strategy + +0:11:29.250,0:11:31.370 +Auto configuration + +0:11:31.370,0:11:32.770 +is what happens + +0:11:32.770,0:11:36.619 +typically or historically I guess you +could say as the system boots. + +0:11:36.619,0:11:39.500 +so all that stuff that comes out about + +0:11:39.500,0:11:40.810 +what + +0:11:40.810,0:11:43.550 +hardwares are on the machine and how it's all +interconnected + +0:11:43.550,0:11:47.350 +all of that is tied up in auto configuration + +0:11:47.350,0:11:50.040 +and that used to happen just once it boots + +0:11:50.040,0:11:52.000 +but in modern systems today + +0:11:52.000,0:11:55.839 +it's an ongoing process. It happens at boot +but it also happens + +0:11:55.839,0:12:00.550 +anytime you plug a new I/O device, a +PCMCIA card, + +0:12:00.550,0:12:03.680 +or you remove a disk or you put in a new disk. + +0:12:03.680,0:12:07.010 +or any sort of activity that changes the I/O + +0:12:07.010,0:12:08.360 +structure of the machine + +0:12:08.360,0:12:10.870 +auto configuration has to get fired back up + +0:12:10.870,0:12:13.050 +and figure out what's disappeared + +0:12:13.050,0:12:18.330 +and cleanup and figure out what new has arrived +to configure it in. + +0:12:18.330,0:12:19.320 +and then we'll talk + +0:12:19.320,0:12:23.870 +a little bit about the configuration of the +device driver + +0:12:23.870,0:12:27.390 +this actually gets into an area that + +0:12:27.390,0:12:28.660 +is + +0:12:28.660,0:12:33.440 +one well let me just give it as a bit +of advice to the class especially those of + +0:12:33.440,0:12:36.780 +you who work in system administration. + +0:12:36.780,0:12:42.010 +You really want to be careful that +you don't learn too much about device drivers + +0:12:42.010,0:12:44.670 +because there is really these three things that + +0:12:44.670,0:12:48.580 +it's not good to learn about and if you do +learn about it it's really good to keep it + +0:12:48.580,0:12:49.740 +to yourself + +0:12:49.740,0:12:51.949 +because if you become an expert or + +0:12:51.949,0:12:54.960 +viewed as an expert in any of these areas + +0:12:54.960,0:12:59.370 +you will become the designated stuccy for +that and your site you'll never get to do + +0:12:59.370,0:13:01.760 +anything + +0:13:01.760,0:13:02.610 +but that + +0:13:02.610,0:13:07.360 +so The three things that I highly +recommend not learning very much about are + +0:13:07.360,0:13:09.060 +device drivers, + +0:13:09.060,0:13:12.320 +send mail configuration files + +0:13:12.320,0:13:13.970 +or anything having to do + +0:13:13.970,0:13:19.350 +with LDAP or anything in +that general domain + +0:13:19.350,0:13:22.660 +because as I say + +0:13:22.660,0:13:24.900 +that will become your life's work + +0:13:24.900,0:13:25.920 +and + +0:13:25.920,0:13:32.920 +there's other things that you might find more interesting. +""Do you have a question?"" + +0:13:33.870,0:13:36.659 +so one of my students empathizes with my point + +0:13:36.659,0:13:39.640 +I believe you said you worked on that mail +system + +0:13:39.640,0:13:43.120 +so you you might know something about +Sendmail configuration files but you don't + +0:13:43.120,0:13:47.850 +have to answer that + +0:13:47.850,0:13:52.100 +okay so we're going to talk about what a device +driver does and really just sort of the entry + +0:13:52.100,0:13:53.170 +points to it + +0:13:53.170,0:13:57.180 +but we're not going to talk about how you +write such a thing, how you debug such a thing + +0:13:57.180,0:14:01.490 +or much of anything about it. I actually used +to teach an entire class believe it or not + +0:14:01.490,0:14:02.720 +about device drivers + +0:14:02.720,0:14:05.849 +but then I realized the error of my ways and I have +since + +0:14:05.849,0:14:12.580 + gone through and made a point of forgetting +every slide in that talk. + +0:14:12.580,0:14:16.860 +okay so then we will move on to File system + +0:14:16.860,0:14:21.540 +and as always we'll start at the high level +talk about the interface what is it that is + +0:14:21.540,0:14:23.020 +exported out of the system + +0:14:23.020,0:14:27.840 +and then we will start diving down in the C and +how do we go about implementing that + +0:14:27.840,0:14:29.010 +so + +0:14:29.010,0:14:31.010 +we'll start with the + +0:14:31.010,0:14:32.560 +so called + +0:14:32.560,0:14:33.680 +Block I/O system + +0:14:33.680,0:14:36.140 +it's historically been called buffer +cache + +0:14:36.140,0:14:38.590 +and you still hear it called that periodically + +0:14:38.590,0:14:42.720 +and the fact of the matter is that there isn't really +about buffer cache anymore, there is just one big + +0:14:42.720,0:14:44.620 +cache in it.Its the VM cache + +0:14:44.620,0:14:47.810 +and the Filesystem has a view into it +and + +0:14:47.810,0:14:50.829 +the processes have a view into it but at +the end of the day + +0:14:50.829,0:14:54.660 +you really don't want the same information +on two different + +0:14:54.660,0:14:56.030 +pages of memory + +0:14:56.030,0:14:59.390 +because that just leads to trouble. + +0:14:59.390,0:15:03.390 +But Filesystems think they have buffers and so +there's this maneuver where we make + +0:15:03.390,0:15:06.149 +these things that look like what historically +were buffers + +0:15:06.149,0:15:08.830 +that really just map into VM system + +0:15:08.830,0:15:11.720 +but they're still managed in the way that +they have been + +0:15:11.720,0:15:15.020 +managed historically + +0:15:15.020,0:15:20.670 +okay We will then get down into Filesystem implementation +the local file system if you will + +0:15:20.670,0:15:23.400 +and into also + +0:15:23.400,0:15:25.730 +soft updates and snapshots. + +0:15:25.730,0:15:26.440 + this + +0:15:26.440,0:15:31.100 +for the time being is something that you see +only in FreeBSD + +0:15:31.100,0:15:35.310 +the alternative to soft updates is journalling +which is %uh more commonly used + +0:15:35.310,0:15:39.630 +for example what is used by ext3 + +0:15:39.630,0:15:41.179 +and so I'll go through soft updates and + +0:15:41.179,0:15:45.260 +a lot of the issues in soft updates are the +same issues that you have to deal with journalling + +0:15:45.260,0:15:48.370 +what is it that we're protecting and how do we +go about doing that + +0:15:48.370,0:15:51.150 +and the difference is in the detail. + +0:15:51.150,0:15:54.630 +There is actually a paper in the back to your +notes if this is something that interests + +0:15:54.630,0:15:55.240 +you + +0:15:55.240,0:15:59.930 +it's a comparison of journalling versus +soft updates that was done + +0:15:59.930,0:16:02.120 +about five or eight years ago. + +0:16:02.120,0:16:08.460 +and not to spoil the punch line but the answers +they both work about are the same + +0:16:08.460,0:16:12.500 +Okay snapshots again is something that +if + +0:16:12.500,0:16:15.920 +you've worked with things like the network +appliance box you're probably quite + +0:16:15.920,0:16:19.640 +aware of what snapshots are and how they do +or don't work for you + +0:16:19.640,0:16:21.959 +this is the same functionality + +0:16:21.959,0:16:27.380 +in the Filesystem implemented in a +somewhat different way + +0:16:27.380,0:16:28.449 +okay so this + +0:16:28.449,0:16:31.940 +Week 6 is really going to be the local +file system + +0:16:31.940,0:16:34.750 +the disk connected to the machine +that we are dealing with. + +0:16:34.750,0:16:39.140 +Week 7 then we get into multiple +Filesystem support so how do we abstract out that + +0:16:39.140,0:16:41.190 +Filesystem layer + +0:16:41.190,0:16:46.430 +and support Multiple Filesystems at the +same time so for example in FreeBSD + +0:16:46.430,0:16:50.199 +you can of course run with their traditional +fast Filesystem + +0:16:50.199,0:16:54.540 +but if you happen to like the Linux Filesystem +better or you have to share a disk + +0:16:54.540,0:16:55.690 +with a Linux machine + +0:16:55.690,0:16:58.310 +you can run the ext2 or ext3 + +0:16:58.310,0:17:01.020 +and it will perfectly happily do that + +0:17:01.020,0:17:01.620 +so + +0:17:01.620,0:17:05.589 +we will have to look then at how do we provide +interface so that we can plug in all these different + +0:17:05.589,0:17:09.260 +Filesystems that we want to support + +0:17:09.260,0:17:12.250 +another area of which there's been a great + +0:17:12.250,0:17:15.309 +deal of growth at least in code complexity + + +0:17:15.309,0:17:17.840 +is so-called Volume Management + +0:17:17.840,0:17:19.370 +so in the + +0:17:19.370,0:17:24.480 +good old days a Filesystem lived on a disk or +piece of disk and that was that + +0:17:24.480,0:17:26.130 +but in this day and age + +0:17:26.130,0:17:31.150 +that won't do any more so we aggregate disks +together by striping them or RAID + +0:17:31.150,0:17:31.980 +arraying them + +0:17:31.980,0:17:33.380 +or various other things + +0:17:33.380,0:17:39.210 +and we need a whole layer in the system just to +manage those disks + +0:17:39.210,0:17:44.280 +we'll then get to the as an example of an alternative +Filesystem we're going to talk about the + +0:17:44.280,0:17:46.530 +Network Filesystem or NFS + +0:17:46.530,0:17:48.500 +but that's not because this is + +0:17:48.500,0:17:51.090 +the world's best remote file system + +0:17:51.090,0:17:55.240 +or the cleanest design or any of the +properties you might hope that + +0:17:55.240,0:17:57.049 +such a class as this one would have + +0:17:57.049,0:17:58.600 +but it's ubiquitous + +0:17:58.600,0:18:00.210 +very widely used + +0:18:00.210,0:18:01.350 +and + +0:18:01.350,0:18:06.850 +so we're going to talk about that one + +0:18:06.850,0:18:07.740 +okay we'll + +0:18:07.740,0:18:10.970 +then once again switch gears in Week 8 + +0:18:10.970,0:18:17.120 +and turn our attention to of Networking and +Interprocess communication + +0:18:17.120,0:18:18.200 +and + +0:18:18.200,0:18:23.210 +again we'll start from the very top so we'll +go through, we'll go with concepts, the terminology + +0:18:23.210,0:18:24.450 +that gets used + +0:18:24.450,0:18:30.230 +and what's the difference between domain +based addressing and an address domain you know + +0:18:30.230,0:18:30.910 +we'll go through + +0:18:30.910,0:18:34.910 + what the basic IPC services are, + +0:18:34.910,0:18:39.080 +essentially what are all the system calls that +have anything to do with networking + +0:18:39.080,0:18:40.590 +and + +0:18:40.590,0:18:43.720 +just sort of describe what each of them are +and I'm going to go through + +0:18:43.720,0:18:45.830 +a somewhat contrived example + +0:18:45.830,0:18:49.840 +that makes use of every one of those interfaces + +0:18:49.840,0:18:52.860 +and just to sort of show how they all connect +together + +0:18:52.860,0:18:54.169 +and for those of you that work + +0:18:54.169,0:18:57.400 +in networking or had done any kind of network +programming + +0:18:57.400,0:19:00.480 +if you're looking for a week to miss and the +Week 8 is the one to miss that's 'cause that's + +0:19:00.480,0:19:02.780 +the sort of most basic + +0:19:02.780,0:19:04.210 +lecture that I'm going to give + +0:19:04.210,0:19:07.910 +If you are not sure whether or not you need to +go through that, there is + +0:19:07.910,0:19:09.540 +one of the papers in the back + +0:19:09.540,0:19:12.620 +it is an introduction to Interprocess communication + +0:19:12.620,0:19:18.279 +read that paper if you say yeah yeah yeah +yeah yeah you are done with Week 8. + +0:19:18.279,0:19:20.590 +on the other hand if you don't come to Week +8 + +0:19:20.590,0:19:22.790 +and then in Week 9 I say + +0:19:22.790,0:19:26.860 +I call on you and say alright what is it + +0:19:26.860,0:19:30.560 +that listen system call does and you +can't tell me + +0:19:30.560,0:19:32.610 +you're gonna get a demerit + +0:19:32.610,0:19:34.340 +okay + +0:19:34.340,0:19:37.770 +then in Week 9 we will get into the actual + +0:19:37.770,0:19:41.419 +networking implementation itself, we go +through system layers as we did + +0:19:41.419,0:19:43.310 +in all the other areas + +0:19:43.310,0:19:44.130 +and + +0:19:44.130,0:19:48.330 +we will spend a significant portion of that +class talking about routing + +0:19:48.330,0:19:50.230 +routing + +0:19:50.230,0:19:53.610 +for those of you that haven't had the pleasure +of dealing with it + +0:19:53.610,0:19:55.540 +is a black art + +0:19:55.540,0:19:58.050 +or at least a dark science + +0:19:58.050,0:19:59.170 +and + +0:19:59.170,0:19:59.930 +so + +0:19:59.930,0:20:02.490 +we'll talk about it + +0:20:02.490,0:20:06.270 +from the perspective first of all of what +do we do locally within the machine + +0:20:06.270,0:20:10.090 +and then what are some of the bigger strategies +that we can use for doing routing + +0:20:10.090,0:20:11.910 +enterprise + +0:20:11.910,0:20:14.840 +wide routing or + +0:20:14.840,0:20:20.190 +area wide routing something like throughout the +state of California or throughout the US whatever + +0:20:20.190,0:20:25.379 +this again like device drivers is really +just sort of a nickel + +0:20:25.379,0:20:26.480 +tour through the + +0:20:27.800,0:20:31.820 +what the choices are what that the basic +strategies are that are used + +0:20:31.820,0:20:33.989 +If you're thinking you're going to walk out +of here + +0:20:33.989,0:20:36.110 +knowing how to set up a routing well sorry + +0:20:36.110,0:20:38.430 +we are not going to get that far + +0:20:38.430,0:20:41.559 +but you should at least have a pretty good idea +of what the issues are + +0:20:41.559,0:20:44.430 +and what the general solutions are + +0:20:44.430,0:20:48.950 +okay then finally in Week 10 well not finally +but next few weeks and + +0:20:48.950,0:20:52.380 +we will go through the Internet Protocols + +0:20:52.380,0:20:54.320 +primarily TCP/IP + +0:20:54.320,0:20:56.560 +and this is + +0:20:56.560,0:20:58.809 +what are the algorithms that are used + +0:20:58.809,0:21:01.030 +and I'm putting a particular emphasis + +0:21:01.030,0:21:03.050 +for this particular class + +0:21:03.050,0:21:05.080 +on + +0:21:05.080,0:21:07.730 +changes that have been made in the protocols + +0:21:07.730,0:21:14.310 +to deal with a lot of the sort of attacks that +we've been seeing the SYN attacks and + +0:21:14.310,0:21:16.880 +that sort of thing + +0:21:16.880,0:21:19.440 +rather than just a straight + +0:21:19.440,0:21:22.440 +iteration of what the actual protocols +are + +0:21:22.440,0:21:24.940 +I'll talk primarily about IPv4 + +0:21:24.940,0:21:31.940 +but I will also try and talk a bit about +IPv6 as well + +0:21:33.510,0:21:35.850 +all right so the first ten weeks are + +0:21:35.850,0:21:38.100 +sort of the kernel course + +0:21:38.100,0:21:40.800 +now we attack two weeks at the end + +0:21:40.800,0:21:42.010 +to talk about + +0:21:42.010,0:21:43.990 +sort of the bigger picture of + +0:21:43.990,0:21:48.240 +System Tuning,Crash dump analysis that level of +thing + +0:21:48.240,0:21:52.940 +The idea is to really consolidate what +we figured out or talked about in the first + +0:21:52.940,0:21:54.710 +ten weeks and + +0:21:54.710,0:21:58.760 +how that applies to tools that we have available +to us to + +0:21:58.760,0:22:00.760 +look at what the system is doing, + +0:22:00.760,0:22:02.649 + analyze what the system is doing + +0:22:02.649,0:22:03.650 +and hopefully + +0:22:03.650,0:22:04.720 +improve + +0:22:04.720,0:22:07.130 +the performance of what the system is doing + +0:22:07.130,0:22:07.750 +and + +0:22:07.750,0:22:12.169 +for the most part the kind of tuning that I'm +talking about is not + +0:22:12.169,0:22:14.740 +going in and hack hack hacking your kernel + +0:22:14.740,0:22:16.510 +because the fact that the matter is + +0:22:16.510,0:22:18.600 +most of the time you can't do that anyway + +0:22:18.600,0:22:22.340 +so it's more looking at it from the perspective +of saying + +0:22:22.340,0:22:26.390 +is this system running badly because it doesn't +have enough memory on it? + +0:22:26.390,0:22:29.470 +or is it running badly because there isn't enough +I/O capacity? + +0:22:29.470,0:22:33.549 +or is it running badly because it's got +enough I/O capacity but + +0:22:33.549,0:22:35.940 +certain drives are being overloaded + +0:22:35.940,0:22:37.309 +or is it + +0:22:37.309,0:22:42.220 +being overrun because we're simply trying +to do too much on this machine? etc. + +0:22:42.220,0:22:45.440 +so that's the sort of level of thing that we're +looking at it + +0:22:45.440,0:22:47.080 +but tied into + +0:22:47.080,0:22:52.130 +lot of concepts that we talked before so we can talk +about active virtual memory + +0:22:52.130,0:22:53.710 +and what that means + +0:22:53.710,0:22:55.120 +and + +0:22:55.120,0:22:58.750 +essentially measure what it is and hopefully +then you will understand in the context of what + +0:22:58.750,0:23:00.690 +we talked about in the VM section + +0:23:00.690,0:23:03.990 +what that really means + +0:23:03.990,0:23:07.460 +the Crash dump analysis is one of these +topics that + +0:23:07.460,0:23:08.730 +you are gonna love or hate + +0:23:08.730,0:23:12.530 +you actually have to deal with crashed +dumps + +0:23:12.530,0:23:13.679 +its people find it invaluable + +0:23:13.679,0:23:15.580 +and if you don't have to deal with Crash dumps + +0:23:15.580,0:23:18.790 +it's an incredible mass of boring detail + +0:23:18.790,0:23:23.240 +the only good part of it is that that's the +whole session is only about an hour long + +0:23:23.240,0:23:25.529 +If it interests you, listen closely + +0:23:25.529,0:23:28.950 +and if it bores you, well, its only an hour long + +0:23:28.950,0:23:32.880 +okay lastly we'll talk a little bit about +security issues + +0:23:32.880,0:23:36.250 +again this is really more to the tools that +are available + +0:23:36.250,0:23:40.750 +to deal with security staff as opposed to a +complete tutorial on + +0:23:40.750,0:23:45.120 +how to implement security so those of you +that deal with security + +0:23:45.120,0:23:48.400 +this is just gonna to be sort of security one oh +one + +0:23:48.400,0:23:50.029 +for those of you + +0:23:50.029,0:23:51.500 +that have but + +0:23:51.500,0:23:54.399 +you'll have to deal with it but haven't really +thought about it + +0:23:54.399,0:23:58.549 +it'll probably scare you to death and +you wonder how to keep the machines from + +0:23:58.549,0:24:02.840 +being hijacked everyday + +0:24:02.840,0:24:08.030 +Okay so that's in essence what we're going +to try and do here + +0:24:08.030,0:24:15.030 +anybody have any comments, questions, thoughts. +No? All right well. + +0:24:16.130,0:24:17.840 +Let's get started + +0:24:17.840,0:24:22.180 +we will be begin on page fifteen with an +overview of the kernel. + +0:24:22.180,0:24:26.040 +Hopefully nobody's lost yet. + +0:24:26.040,0:24:29.310 +What's a kernel? All right. + +0:24:29.310,0:24:31.370 +so starting at the very top + +0:24:31.370,0:24:33.070 +the big broad brush + +0:24:33.070,0:24:35.140 +what we have is + +0:24:35.140,0:24:38.330 +a UNIX virtual machine and + +0:24:38.330,0:24:41.660 + virtual machines are actually something +that has been around + +0:24:41.660,0:24:44.539 +as a concept since the sixties + +0:24:44.539,0:24:48.919 + difference is really just sort of the level +of the interface that people have dealt with + +0:24:48.919,0:24:51.360 +when they talk about Virtual Machines + +0:24:51.360,0:24:53.610 +in the 1960s + +0:24:53.610,0:24:56.770 +computers were these enormous things you would +have + +0:24:56.770,0:24:58.870 +your computer room would be something that'd be + +0:24:58.870,0:25:01.909 +three times the size of this conference +room if you had + +0:25:01.909,0:25:03.230 +a computer + +0:25:03.230,0:25:05.530 +the computer itself was + +0:25:05.530,0:25:07.840 +tall as a refrigerator freezer + +0:25:07.840,0:25:08.950 +imagine + +0:25:08.950,0:25:13.909 +five or eight or ten of these units +side by side that itself made up the computer + +0:25:13.909,0:25:16.080 +that would be one big + +0:25:16.080,0:25:20.030 +for the core processor and the one which +should be the floating point unit and several + +0:25:20.030,0:25:24.080 +of them that would be the memory the core memory +literally the core memory + +0:25:24.080,0:25:29.110 +and then they'd be other rows of these +disk drives which were about the size of the washing + +0:25:29.110,0:25:29.660 +machine + +0:25:29.660,0:25:34.169 +and then behind that since you couldn't store +everything on disks so + +0:25:34.169,0:25:36.300 +then you had rows of tape drives + +0:25:36.300,0:25:37.880 +and then you had this little + +0:25:37.880,0:25:39.610 +set of sort of + +0:25:39.610,0:25:43.330 +munchkins that would run around and and tend +to the machine and they'd mount tapes and take + +0:25:43.330,0:25:46.710 +off tapes and mount disc packs and remove disc packs +because + +0:25:46.710,0:25:49.760 +the drives themselves were very expensive and +so + +0:25:49.760,0:25:53.110 +you wouldn't just as today we have a + + +0:25:53.110,0:25:56.090 +one spindle that was dedicated just to one set +of platters + +0:25:56.090,0:25:57.130 +you could take out a + +0:25:57.130,0:25:59.460 +set of platters and put in another + +0:25:59.460,0:26:02.540 +hundred megabytes set of platters and these are +platters that are + +0:26:02.540,0:26:05.280 +this big around and it's like six or eight +of them and + +0:26:05.280,0:26:09.140 + giant head assemblies they comes rumbling in and +out + +0:26:09.140,0:26:12.440 +anyway one of these giant giant machines + +0:26:12.440,0:26:17.380 +that costs many millions of dollars would run +at about ten + +0:26:17.380,0:26:21.120 +million instructions per second, 10 mips + +0:26:21.120,0:26:21.630 +and 10 mips + +0:26:21.630,0:26:28.330 + was more computing power than anybody +could possibly imagine using in a single application + +0:26:28.330,0:26:28.880 +just + +0:26:28.880,0:26:31.050 +by contrast you know this + +0:26:31.050,0:26:34.070 +four-year-old laptop here is probably on +the order of + +0:26:34.070,0:26:36.440 +one or two hundred mips + +0:26:36.440,0:26:37.140 +but anyway + +0:26:37.140,0:26:40.760 +people couldn't really view what we would +do with a lot of computing power + +0:26:40.760,0:26:44.640 +and the other thing was that you didn't have +a notion of sort of an operating system that had + +0:26:44.640,0:26:45.890 +applications running on it + +0:26:45.890,0:26:46.760 +because + +0:26:46.760,0:26:50.160 +everybody wanted to write straight to +the raw hardware + +0:26:50.160,0:26:51.750 +and so + +0:26:51.750,0:26:55.900 +what IBM who was a big manufacturer +of machines in those days + +0:26:55.900,0:26:59.060 +did what they came up with this thing called +the VM + +0:26:59.060,0:27:00.770 +and this was a little + +0:27:00.770,0:27:02.549 +you'd call an operating system really + +0:27:02.549,0:27:05.130 +but what it did is it cloned + +0:27:05.130,0:27:09.270 +independent copies of the machine that worked just +like the original machines so you could boot + +0:27:09.270,0:27:11.769 +something that you thought it was an operating +system + +0:27:11.769,0:27:13.380 +on top of VM + +0:27:13.380,0:27:16.750 +so you take one least ten mip machines and +it would clone + +0:27:16.750,0:27:20.050 +six identical one mip copies + +0:27:20.050,0:27:22.030 +and then you could boot + +0:27:22.030,0:27:24.700 +whatever you wanted on each one of those machines +so + +0:27:24.700,0:27:29.510 +if you were doing database stuff you would boot your +database because database cannot ran on the raw hardware + +0:27:29.510,0:27:32.920 +or if you're doing payroll who would boot up the payroll +program + +0:27:32.920,0:27:37.950 +or if you actually tried to service +users you could boot a time sharing batch thing + +0:27:37.950,0:27:40.790 +that would read card images and print +stuff out + +0:27:40.790,0:27:44.460 +or they even had TSO the Time Sharing +Option where you could interactively sit + +0:27:44.460,0:27:45.559 +and type and send + +0:27:45.559,0:27:47.560 +stuffs in and get answers back + +0:27:47.560,0:27:48.570 + and + +0:27:48.570,0:27:51.429 +also you could boot TSO so whatever set +of + +0:27:51.429,0:27:52.219 + + +0:27:52.219,0:27:55.339 +things you need you could boot them and they ran +independently as if they were running on their + +0:27:55.339,0:27:56.470 +own machine + +0:27:56.470,0:28:03.150 +but all the VM did was it give you an exact +raw copy of the hardware + +0:28:03.150,0:28:04.529 +so when UNIX came along + +0:28:04.529,0:28:07.350 +they sort of liked the notion of + +0:28:07.350,0:28:11.509 +providing the concept of independent +things that you could operate in + +0:28:11.509,0:28:13.610 +but they wanted it at a higher level + +0:28:13.610,0:28:15.610 +so you're looking really to do it + +0:28:15.610,0:28:17.480 +instead of at the raw hardware level + +0:28:17.480,0:28:19.679 +to do it at a process level + +0:28:19.679,0:28:23.799 +and the idea that then was that the interface you +would program to would be what we think of as + +0:28:23.799,0:28:26.090 +a System call interface today + +0:28:26.090,0:28:27.849 +and the idea then was that + +0:28:27.849,0:28:30.740 +you would be given a process or set of processes + +0:28:30.740,0:28:34.990 +and those were independent. your process +couldn't affect + +0:28:34.990,0:28:38.830 +the address space of another processor. You couldn't reach +over and mess around with their addresses, + +0:28:38.830,0:28:41.030 +you couldn't mess around with their I/O +channels + +0:28:41.030,0:28:43.179 +you could slow them down by + +0:28:43.179,0:28:44.299 +being a pig but + +0:28:44.299,0:28:47.980 +that was about the only way that you could affect +other processes + +0:28:47.980,0:28:48.480 +and + +0:28:48.480,0:28:49.830 +so + +0:28:49.830,0:28:52.669 +what the interfaces that they had there + +0:28:52.669,0:28:58.660 +was one that had these characteristics +had a paged virtual address space + +0:28:58.660,0:29:02.980 +so you didn't have to know as in the old days how much physical +memory is on the machine and make your application + +0:29:02.980,0:29:04.740 +fit into that amount of memory + +0:29:04.740,0:29:07.950 +you just had what looked like a large + +0:29:07.950,0:29:11.710 +uniform address space even if the underlying +hardware had segments or some other + +0:29:11.710,0:29:13.580 +hardware brain damage + +0:29:13.580,0:29:17.390 +it looked to you like he just had a big uniform +address space and + +0:29:17.390,0:29:21.070 +the size of your address space was independent +of the amount of memory that was on your machine + +0:29:21.070,0:29:23.900 +your address space couldn't be bigger than amount of +physical memory + +0:29:23.900,0:29:26.499 +cause we sort of move pages around underneath + +0:29:26.499,0:29:29.320 +whatever part address space was actually +active + +0:29:29.320,0:29:34.260 +and there's obviously limits to this if +you are trying to run a 1 gigabyte of + +0:29:34.260,0:29:35.630 +application on top of + +0:29:35.630,0:29:37.240 +ten megabytes of memory + +0:29:37.240,0:29:40.880 +it's probably going to bring new meaning to +same day service + +0:29:40.880,0:29:45.519 +but if you're willing to wait long enough it +will eventually move the pages around and you will + +0:29:45.519,0:29:49.740 +progress through getting your application run + +0:29:49.740,0:29:53.890 +another thing was dealing with software +interrupts + +0:29:53.890,0:29:55.789 +in the old days + +0:29:55.789,0:29:58.749 +you had to understand how the hardware worked + +0:29:58.749,0:30:03.900 +in order to deal with exceptional conditions +so for example if you did a divide by zero + +0:30:03.900,0:30:08.170 +the hardware would jump through some +vector location or + +0:30:08.170,0:30:08.630 +something + +0:30:08.630,0:30:12.799 +and you had know how that worked and make +sure that you had your program + +0:30:12.799,0:30:16.510 +usually some little bit of assembly language +set up to deal with that + +0:30:16.510,0:30:19.870 +and UNIX said let's let's get away +from the hardware here + +0:30:19.870,0:30:22.080 +and so they did this thing called signals + +0:30:22.080,0:30:25.700 +and so they just define a set of the signals is that +if you do divide by zero + +0:30:25.700,0:30:29.529 +you simply register a routine you +want to have called you don't have to know + +0:30:29.529,0:30:31.220 +how the hardware figured it out + +0:30:31.220,0:30:36.740 +you just know that that routine is going to get +called and you can deal with it at that point + +0:30:36.740,0:30:40.960 +well we got set of timers and counters to keep +track of what we're doing, this is really more + +0:30:40.960,0:30:43.490 +for counting than anything else but + +0:30:43.490,0:30:46.970 +applications may want to have access to that. + +0:30:46.970,0:30:51.720 +we have a set of identifiers that we're +going to use for things like accounting, + +0:30:51.720,0:30:54.830 +protection and scheduling and so on + +0:30:54.830,0:30:55.820 +and one of the + +0:30:55.820,0:31:00.320 +the early philosophies of UNIX was to try +and keep it simple. + +0:31:00.320,0:31:02.630 +operating systems have gotten very baroque + +0:31:02.630,0:31:04.490 +in particular the thing that + +0:31:04.490,0:31:07.350 +pre dated UNIX was a thing called +Multix + +0:31:07.350,0:31:12.820 +Multix was was a joint project between +Honeywell, a big computer manufacturer of the + +0:31:12.820,0:31:15.740 +time + +0:31:15.740,0:31:17.129 +AT&T bell laboratories + +0:31:17.129,0:31:19.750 +the big industrial laboratory at that time + +0:31:19.750,0:31:21.380 +and MIT + +0:31:21.380,0:31:23.430 +a big university then and + +0:31:23.430,0:31:24.690 +still today + +0:31:24.690,0:31:29.259 +and those three organizations got +together to try and build this + +0:31:29.259,0:31:31.400 +time sharing operating system + +0:31:31.400,0:31:32.280 +and it + +0:31:32.280,0:31:33.770 +it just got bigger and + +0:31:33.770,0:31:37.160 +more grandiose and more complex and never +finished + +0:31:37.160,0:31:38.979 +because as soon as they sort of see + +0:31:38.979,0:31:42.709 +oh we know how to do that but we could +do this other thing too and so then they would tear it + +0:31:42.709,0:31:43.429 +apart and + +0:31:43.429,0:31:46.440 +they never really got to something that + +0:31:46.440,0:31:48.210 +could be put into production + +0:31:48.210,0:31:49.919 +and so the + +0:31:49.919,0:31:50.570 +AT&T + +0:31:50.570,0:31:54.340 +Bell laboratories decided to pull out of +that project + +0:31:54.340,0:31:55.940 +and + +0:31:55.940,0:32:00.000 +the two of the people that had been working on +that project, Ken Thompson and Dennis Richie + +0:32:00.000,0:32:04.390 +were sort of bummed because they were now +back to typing cards and putting them through + +0:32:04.390,0:32:05.259 +card readers and + +0:32:05.259,0:32:07.960 +they had gotten used to the idea that you could +actually + +0:32:07.960,0:32:11.559 +sit at an ASSR33 teletype and interact +with your computer + +0:32:11.559,0:32:13.440 +and so + +0:32:13.440,0:32:18.230 +they found an old %uh PDP-8 sitting off in +the corner that had been abandoned + +0:32:18.230,0:32:22.120 +and started working on this little tiny operating +system which they called UNIX + +0:32:22.120,0:32:26.549 +which eventually moved to the PDP-11 and +became what we have today + +0:32:26.549,0:32:28.050 +but because it was + +0:32:28.050,0:32:32.120 +they were coming first of all from Multix +where everything had been done and + +0:32:32.120,0:32:34.110 +in great grandiose detail + +0:32:34.110,0:32:37.549 +and because they're fundamentally were two + of them working on it and they wanted to get something + +0:32:37.549,0:32:38.370 +done and + +0:32:38.370,0:32:40.130 +within a year or so + +0:32:40.130,0:32:41.529 +one of their philosophies was + +0:32:41.529,0:32:44.099 +let's find the one way of doing things + +0:32:44.099,0:32:48.180 +let's not have eight ways from Sunday let's just +get the one way + +0:32:48.180,0:32:53.860 +and that's what we will provide. So what is +the sort of core set of things that we need. + +0:32:53.860,0:32:58.620 +well first thing is when it comes to identifiers, +let's not have you know + +0:32:58.620,0:33:00.430 +eighty thousand different identifiers + +0:33:00.430,0:33:03.140 +so they came up with process identifiers, + +0:33:03.140,0:33:09.620 +user identifier and at that time a single group +identifier and later expanded + +0:33:09.620,0:33:14.200 +and they used that sort of identifiers for everything +so its used for counting, used for making + +0:33:14.200,0:33:17.410 +protection decisions, used for scheduling +decisions + +0:33:17.410,0:33:19.470 +and + +0:33:19.470,0:33:24.279 +again it was the simplicity of thing which +was what was driving their decision + +0:33:24.279,0:33:28.840 +but they're really sort of two key ideas +that they had + +0:33:28.840,0:33:30.880 +that really made the difference that + +0:33:30.880,0:33:32.539 +that's what set them up side + +0:33:32.539,0:33:34.749 +from what everybody else had done before them + +0:33:34.749,0:33:35.450 +and which + +0:33:35.450,0:33:39.740 +in retrospect is something that has been pervasive +more or less ever since + +0:33:39.740,0:33:41.869 +the first of these was the notion + +0:33:41.869,0:33:44.840 +that we have a unique descriptor space + +0:33:44.840,0:33:46.289 +that is + +0:33:46.289,0:33:51.250 +given a descriptor it can reference +any I/O device + +0:33:51.250,0:33:53.650 +so or even any kind of I/O channel + +0:33:53.650,0:33:58.270 +so you can have a descriptor for terminal +or descriptor for a file or descriptive for + +0:33:58.270,0:34:02.240 +a disk or descriptor for a pipe or descriptor +for a socket + +0:34:02.240,0:34:03.500 +and + +0:34:03.500,0:34:04.790 +you don't need to know + +0:34:04.790,0:34:07.940 +what it references in order to be able to read +and write that thing + +0:34:07.940,0:34:11.290 +so if I hand you a descriptor +you can read from that the descriptor or you can write + +0:34:11.290,0:34:13.259 +to that descriptor + +0:34:13.259,0:34:15.189 +and + +0:34:15.189,0:34:17.359 +the correct thing will happen + +0:34:17.359,0:34:19.089 +and you'd say well + +0:34:19.089,0:34:23.629 +that's so obvious I mean how else could you +possibly think of doing it? + +0:34:23.629,0:34:25.179 +well predating UNIX + +0:34:25.179,0:34:28.059 +everything was done with + +0:34:28.059,0:34:29.379 +a little subsystem + +0:34:29.379,0:34:33.419 +that would open a file, read a file, write a +file, close a file + +0:34:33.419,0:34:37.429 +and there was another set of system calls which +would open a terminal, read a terminal, write terminal, + +0:34:37.429,0:34:38.089 +close terminal + +0:34:38.089,0:34:39.210 +and yet another one + +0:34:39.210,0:34:42.409 +which was create a pipe, read a pipe, +write a pipe and so on. + +0:34:42.409,0:34:47.699 +so if you are just a drop dead stupid +program like say CAD + +0:34:47.699,0:34:51.579 +you would have to have code in there and say was +my input a terminal which in case I need to + +0:34:51.579,0:34:53.159 +use the read terminal + +0:34:53.159,0:34:57.419 +or is it a file which in case I need +to use read file or is it a pipe in which in case + +0:34:57.419,0:34:59.189 +I need to use read pipe + +0:34:59.189,0:35:01.860 +and so the program itself had to have all +this + +0:35:01.860,0:35:02.859 +coding in it + +0:35:02.859,0:35:04.409 +whereas when they went to + +0:35:04.409,0:35:07.159 +the uniform descriptor space + +0:35:07.159,0:35:09.630 +CAD doesn't know it doesn't need to know +it just says + +0:35:09.630,0:35:10.819 +read my input, + +0:35:10.819,0:35:13.979 +write the output + +0:35:13.979,0:35:17.059 +and it works and we add a new type of descriptor + +0:35:17.059,0:35:17.600 +and + +0:35:17.600,0:35:21.700 +CAD just continues to work just as it always +did. + +0:35:21.700,0:35:24.199 +So this proved to be a very powerful construct + +0:35:24.199,0:35:27.019 +and pretty much every operating system after +UNIX + +0:35:27.019,0:35:28.659 +did that there's + +0:35:28.659,0:35:30.210 +one exception of %uh + +0:35:30.210,0:35:32.549 +large company in the Pacific North-West + +0:35:32.549,0:35:35.830 +that still has not quite uniform descriptor +space + +0:35:35.830,0:35:38.380 +but %uh that's part of their legacy that really + +0:35:38.380,0:35:39.900 +they're working on that. + +0:35:39.900,0:35:42.009 +Longhorn will be here. + +0:35:42.009,0:35:43.939 +and anyway + +0:35:43.939,0:35:46.190 +this set of facilities then + +0:35:46.190,0:35:50.150 +makes up the UNIX virtual machine + +0:35:50.150,0:35:51.559 +and + +0:35:51.559,0:35:55.559 +in some sense we still see virtual machines +being used today in fact we're seeing sort + +0:35:55.559,0:35:56.749 +of a reversion + +0:35:56.749,0:36:01.429 +back to some of the IBM stuff in things +like the VMware + +0:36:01.429,0:36:03.079 +which is + +0:36:03.079,0:36:07.029 +essentially allow you to go back to booting +native operating systems again so sort of + +0:36:07.029,0:36:08.280 +interesting to watch + +0:36:08.280,0:36:09.060 +that the sort of + +0:36:09.060,0:36:12.919 +pendulum of back going back and forth +of what's the correct layer + +0:36:12.919,0:36:14.609 +for for doing + +0:36:14.609,0:36:18.890 +virtual machines + +0:36:18.890,0:36:22.499 +Okay? so far so good? + +0:36:22.499,0:36:24.719 +all right so I said that there were + +0:36:24.719,0:36:27.160 +two key ideas that UNIX had + +0:36:27.160,0:36:30.279 +the first of these being the uniform descriptor +space + +0:36:30.279,0:36:35.819 +the second one which was really critical was +this notion of processes as a commodity + +0:36:35.819,0:36:37.309 +item + +0:36:37.309,0:36:40.220 +so here on Page 17 I've tried to lay +it out + +0:36:40.220,0:36:41.090 +the + +0:36:41.090,0:36:44.159 +that the components that make up a process + +0:36:44.159,0:36:45.759 +and + +0:36:45.759,0:36:50.359 +what do I really mean when I say a process as +a commodity item + +0:36:50.359,0:36:53.650 +okay leading up to + +0:36:53.650,0:36:54.689 +UNIX + +0:36:54.689,0:36:56.800 +the systems that pre-dated it, + +0:36:56.800,0:36:59.200 +processes were these very large + +0:36:59.200,0:37:02.169 +heavyweight expensive things + +0:37:02.169,0:37:02.779 +and + +0:37:02.779,0:37:04.539 +if you look at + +0:37:04.539,0:37:08.629 +MVS which was the operating system +that ran on IBM for doing multiple processing + +0:37:08.629,0:37:10.509 +and + +0:37:10.509,0:37:13.799 +the system administrator would decide at boot +time + +0:37:13.799,0:37:17.019 +what degree of multiprocessing they wish +to support + +0:37:17.019,0:37:18.140 +so they'd say well + +0:37:18.140,0:37:20.739 +well, we'll let upto six things happen at once + +0:37:20.739,0:37:22.490 +and so as part of booting up + +0:37:22.490,0:37:24.419 +they would create six + +0:37:24.419,0:37:25.349 +processes + +0:37:25.349,0:37:30.059 +and now you as a user if you wanted to do +something let's say you wanted to + +0:37:30.059,0:37:32.009 +compile and run a program + +0:37:32.009,0:37:34.960 +you would be given a process + +0:37:34.960,0:37:36.019 +and it was up to you + +0:37:36.019,0:37:39.369 +to figure out how to stage what you needed +done + +0:37:39.369,0:37:39.819 +and + +0:37:39.819,0:37:43.930 +that this was often fairly complex + +0:37:43.930,0:37:47.880 +and so you would have to write out all the +steps that you wanted + +0:37:47.880,0:37:50.300 +in this wonderful thing called JCL + +0:37:50.300,0:37:52.259 +Job Control Language. + +0:37:52.259,0:37:56.650 +Job Control Language was send mail configuration +file of the sixties + +0:37:56.650,0:38:00.679 +there where people whose sole job at the company +was how to put this stuff together 'cause + +0:38:00.679,0:38:04.189 +all you had to do is get one extra space or +a missing comma + +0:38:04.189,0:38:05.000 +something in there + +0:38:05.000,0:38:08.630 +and the whole thing would just blow up. it would +just sort of spit the card deck back at + +0:38:08.630,0:38:09.799 +you and say well + +0:38:09.799,0:38:13.500 +somewhere in there is a mistake that's sort of +in the general area of this card + +0:38:13.500,0:38:15.549 +and I can't deal with it. Fix it. + +0:38:15.549,0:38:16.489 +and of course + +0:38:16.489,0:38:20.550 +in those days it wasn't just a matter of hitting +carriage when you know make carriage return you have to + +0:38:20.550,0:38:25.239 +get your deck pull out the card, and type the +new one, put it back in and re-submit it + +0:38:25.239,0:38:28.729 +As heaven forbid you couldn't touch that +card reader you know, it had to be done by + +0:38:28.729,0:38:29.970 +an operator + +0:38:29.970,0:38:32.869 +so the card deck will read through it would +disappear and + +0:38:32.869,0:38:36.800 +you know if you're lucky a few minutes later +if you were not lucky a few hours later + +0:38:36.800,0:38:37.849 +you would get + +0:38:37.849,0:38:39.570 +a print out + +0:38:39.570,0:38:43.419 +which was what had happened and then you could +look at it and you know + +0:38:43.419,0:38:47.209 +I put a comma in the wrong place I guess +I get to do it all again + +0:38:47.209,0:38:49.930 +so + +0:38:49.930,0:38:54.940 +the thing you would need to do there for compiling and running a program + +0:38:54.940,0:38:59.579 +was you'd have to break into these steps. well +I need to run the preprocessor + +0:38:59.579,0:39:04.670 +and so clean out whatever gump that was left +over on that process from the previous user + +0:39:04.670,0:39:06.240 +put the preprocessor in there + +0:39:06.240,0:39:10.530 +and then read from this file here let's +say I gotta put it somewhere so creative + +0:39:10.530,0:39:12.510 +scratch file over on this disk and + +0:39:12.510,0:39:17.299 +it was excruciating detail like how many cylinders +and how many tracks and this and that + +0:39:17.299,0:39:19.139 +blocks blah blah blah + +0:39:19.139,0:39:23.119 +and don't forget any of those parameters 'cause +it'll spit it out if you do + +0:39:23.119,0:39:26.890 +and so then it would run the first step in that +if its successful then you'd have sitting + +0:39:26.890,0:39:28.899 +in this scratch file that you had created + +0:39:28.899,0:39:33.100 +the output of the preprocessor and then +you'd load the first pass of the compiler + +0:39:33.100,0:39:36.930 +and you say now read from that scratch file +and create this other scratch file over here and + +0:39:36.930,0:39:39.450 +when thats successful and we need to delete that +one + +0:39:39.450,0:39:43.830 +and then load the second pass, put that back +into another scratch file and then we run this + +0:39:43.830,0:39:45.950 +assembler, and the optimizer then the + +0:39:45.950,0:39:47.750 +loader this and that + +0:39:47.750,0:39:49.410 +finally run the program + +0:39:49.410,0:39:50.900 +and if all goes well + +0:39:50.900,0:39:57.029 +you know at step sixteen out comes the answer + +0:39:57.029,0:39:58.129 +forty two. so UNIX + +0:39:58.129,0:40:00.819 +said, look this is silly + +0:40:00.819,0:40:02.880 +a lot of this is just + +0:40:02.880,0:40:04.310 +bookkeeping + +0:40:04.310,0:40:07.249 +and computers do bookkeeping really well + +0:40:07.249,0:40:12.179 +and you'll recall yeah but it's going to take +all these cycles it's like + +0:40:12.179,0:40:16.309 +computers are supposed to be labor-saving +devices right? so + +0:40:16.309,0:40:20.150 +they came up with this notion that they would +create processes on the fly as needed + +0:40:20.150,0:40:21.159 +you had + +0:40:21.159,0:40:25.549 +you've had a preprocessor in two +steps of the compiler and then + +0:40:25.549,0:40:27.109 +optimizer and then a loader + +0:40:27.109,0:40:29.410 +we just create Boom seven processes + +0:40:29.410,0:40:31.920 +and we connect them together with pipes + +0:40:31.920,0:40:35.180 +and so we take the input and you know run +through in + +0:40:35.180,0:40:38.270 +through the pipes and you know out the end +you get the the + +0:40:38.270,0:40:39.629 +executable + +0:40:39.629,0:40:40.030 +and + +0:40:40.030,0:40:42.880 +we will simply create each of these processes + +0:40:42.880,0:40:44.650 +and + +0:40:44.650,0:40:46.549 +so you as a user just + +0:40:46.549,0:40:49.479 +type you know the C compiler and it just + +0:40:49.479,0:40:52.429 +fork these things pipe them together got the result + +0:40:52.429,0:40:53.640 +and + +0:40:53.640,0:40:57.509 +then once it was done with this processes is +just threw them away so any time you'd create a + +0:40:57.509,0:41:00.479 +new process and it came to you pristine clean + +0:41:00.479,0:41:04.239 +and you needed a bunch of things it did +put everything in intermediate files + +0:41:04.239,0:41:07.549 +the fact of the matter is in the early days + +0:41:07.549,0:41:08.129 +those computers + +0:41:08.129,0:41:11.910 +didn't really have enough memory to support +all that stuff at once so + +0:41:11.910,0:41:15.809 +behind you those pipes were actually implemented +as files + +0:41:15.809,0:41:19.319 +but you didn't have at least to remember to create +them and delete them + +0:41:19.319,0:41:20.200 +and deal with them + +0:41:20.200,0:41:24.020 +as far as you were concerned it just look stuff +flowing through pipes and of course today it + +0:41:24.020,0:41:24.490 +just + +0:41:24.490,0:41:27.989 +does flow through pipes in memory + +0:41:27.989,0:41:29.439 +okay so + +0:41:29.439,0:41:33.689 +this notion then that that we're just gonna +create processes on the fly is needed and + +0:41:33.689,0:41:35.559 +connect them together as needed + +0:41:35.559,0:41:38.039 +it was a novel concept + +0:41:38.039,0:41:43.599 +and it wasn't that somehow mysteriously figured +out how to create processes cheaply + +0:41:43.599,0:41:44.839 +cause they hadn't + +0:41:44.839,0:41:46.180 +they were still + +0:41:46.180,0:41:49.959 +really expensive to create + +0:41:49.959,0:41:52.210 +but that extra effort + +0:41:52.210,0:41:53.029 +was + +0:41:53.029,0:41:56.089 +worth it because it was saving a lot of programming +time + +0:41:56.089,0:41:59.809 +so my favorite example is you run ls + +0:41:59.809,0:42:01.810 +so we have to create a process + +0:42:01.810,0:42:04.259 +load the ls binary into it + +0:42:04.259,0:42:06.180 +it prints a line or two on your screen + +0:42:06.180,0:42:10.609 +and we tear the entire thing down and return +all its resources back to the system + +0:42:10.609,0:42:14.979 +more than ninety percent of the cost of running +ls is creating and destroying the process + +0:42:14.979,0:42:19.239 +a tiny fraction of it is actually running +ls + +0:42:19.239,0:42:24.259 +but it goes so fast, who cares right + +0:42:24.259,0:42:25.749 +so the point is that + +0:42:25.749,0:42:30.039 +that concept of just creating things as +needed + +0:42:30.039,0:42:31.780 +again was very powerful + +0:42:31.780,0:42:35.709 +and is one that is just pervasive today + +0:42:35.709,0:42:38.639 +okay so what is a process actually made up +of + +0:42:38.639,0:42:43.179 +it gets some amount of CPU time or at +least we do dearly hope that it gets some + +0:42:43.179,0:42:46.050 +amount of CPU time, the lack of getting +CPU time + +0:42:46.050,0:42:46.670 +that makes it + +0:42:46.670,0:42:47.979 +a computer so sluggish + +0:42:47.979,0:42:49.409 +of course + +0:42:49.409,0:42:51.920 +others really boils down to scheduling + +0:42:51.920,0:42:54.249 +and we're going to talk about scheduling + +0:42:54.249,0:42:56.279 +probably more than you care to + +0:42:56.279,0:42:59.219 +in a couple weeks time + +0:42:59.219,0:43:01.619 +we have the asynchronous events + +0:43:01.619,0:43:04.569 +these are the external events that + +0:43:04.569,0:43:05.659 +are coming in + +0:43:05.659,0:43:07.679 +so + +0:43:07.679,0:43:10.169 +they may be either things that + +0:43:10.169,0:43:14.339 +were coming in from the outside world like +start, stop and quit + +0:43:14.339,0:43:15.279 +oh + +0:43:15.279,0:43:18.170 +out-of-band data arrival notification that kind +of thing + +0:43:18.170,0:43:22.339 +or it may in fact be things that the program +is bringing down upon itself + +0:43:22.339,0:43:25.590 +such as a segment fault, a divide by zero + +0:43:25.590,0:43:26.910 +and some other + +0:43:26.910,0:43:31.959 +what would normally be viewed as incorrect +operation + +0:43:31.959,0:43:35.849 +and so we'll talk about that when we talk about +signals + +0:43:35.849,0:43:37.039 +every program + +0:43:37.039,0:43:38.899 +gets some amount of memory + +0:43:38.899,0:43:42.659 +it gets an initial amount when it starts +up injured generally allocates more as it + +0:43:42.659,0:43:45.229 +goes along + +0:43:45.229,0:43:49.429 +this of course we will deal with very extensively +will spend an entire week on it + +0:43:49.429,0:43:54.249 +when we talk about how virtual memory is implemented + +0:43:54.249,0:43:54.609 +and + +0:43:54.609,0:43:57.429 +then we get I/O descriptors + +0:43:57.429,0:44:02.259 +I used to say that every program had to have +at least one I/O descriptor since + +0:44:02.259,0:44:04.910 +it absolutely had no input + +0:44:04.910,0:44:06.329 +absolutely no output + +0:44:06.329,0:44:09.049 +then it was sort of pointless + +0:44:09.049,0:44:12.900 +of course I had to have one of my students +come up and point out to me there is an a + +0:44:12.900,0:44:13.849 +class of programs + +0:44:13.849,0:44:16.469 +which don't need I/O descriptors + +0:44:16.469,0:44:17.440 +and that is + +0:44:17.440,0:44:19.549 +these things called benchmarks + +0:44:19.549,0:44:23.249 +it just compute something all we really care +about is how long it takes them to compute + +0:44:23.249,0:44:24.959 +we don't actually care what the answer is + +0:44:24.959,0:44:26.019 +In theory we don't + +0:44:26.019,0:44:29.779 +I personally like my benchmark stop with +something so I can see it there + +0:44:29.779,0:44:31.489 +doing computing the right thing + +0:44:31.489,0:44:33.169 +but in theory + +0:44:33.169,0:44:35.919 +that wouldn't be necessary + +0:44:35.919,0:44:38.650 +outside of that class of programs + +0:44:38.650,0:44:42.670 +everything needs some sort of descriptors and +of course we'll talk about descriptors + +0:44:42.670,0:44:43.659 +quite extensively + +0:44:43.659,0:44:47.349 +as we go through the I/O subsystem + +0:44:47.349,0:44:50.969 +okay so the executive summary is that processes +are + +0:44:50.969,0:44:54.969 +the fundamental service that is provided by +UNIX + +0:44:54.969,0:44:58.430 +and + +0:44:58.430,0:45:02.849 +what we're going to spend essentially the +next two and a half weeks working on + +0:45:02.849,0:45:04.769 +is + +0:45:04.769,0:45:07.079 +what what makes up processes + +0:45:07.079,0:45:10.180 +we'll go into much more detail about each of these +four points + +0:45:10.180,0:45:11.769 +and + +0:45:11.769,0:45:13.630 +then how do we actually go about + +0:45:13.630,0:45:14.390 +providing that + +0:45:14.390,0:45:16.639 +bit of service + +0:45:16.639,0:45:17.900 +the next thing that I'm + +0:45:17.900,0:45:22.210 +going to do now is this go through and lay +out some of the terminology that + +0:45:22.210,0:45:23.239 +we have when + +0:45:23.239,0:45:25.130 +we're talking about processes + +0:45:25.130,0:45:29.229 +so this is sort of the big picture here were +on page eighteen + +0:45:29.229,0:45:30.669 +and + +0:45:30.669,0:45:33.669 +you can see we have sort of three bits that +make up + +0:45:33.669,0:45:36.640 +the system + +0:45:36.640,0:45:39.029 +we have the currently running user process + +0:45:39.029,0:45:41.180 +and then what we call the top half of the kernel + +0:45:41.180,0:45:43.699 +and the bottom half of the kernel + +0:45:43.699,0:45:47.049 +now this would be a picture for a uniprocessor + +0:45:47.049,0:45:49.299 +so one CPU + +0:45:49.299,0:45:51.209 +if we had a multiprocessor + +0:45:51.209,0:45:54.009 +%uh then we would have + +0:45:54.009,0:45:57.130 +one instance of the kernel + +0:45:57.130,0:45:59.529 +but multiple instances of the user process + +0:45:59.529,0:46:02.879 +but for any given CPU on a multiprocessor + +0:46:02.879,0:46:05.709 +it is running exactly one process + +0:46:05.709,0:46:09.309 +so you may think they we're running for four-five +processes all at once + +0:46:09.309,0:46:14.319 +but the fact of the matter is that any instant +in time there's only one process which is + +0:46:14.319,0:46:16.299 +actually running + +0:46:16.299,0:46:18.609 +and + +0:46:18.609,0:46:21.429 +that is the one that we have loaded in the system + +0:46:21.429,0:46:25.199 +now we give the illusion that were running +lots of things because we switch between them + +0:46:25.199,0:46:26.100 +rather quickly + +0:46:26.100,0:46:29.269 +so it looks like things are happening in all +windows at once + +0:46:29.269,0:46:31.430 +but in reality + +0:46:31.430,0:46:33.619 +that's not really happening + +0:46:33.619,0:46:36.440 +okay so there is a set of properties that I want to +look at + +0:46:36.440,0:46:40.899 +that had to do with each one of these parts here + +0:46:40.899,0:46:44.359 +but just to sort of look at it from the +big picture perspective + +0:46:44.359,0:46:45.970 +what you see here + +0:46:45.970,0:46:47.180 +is + +0:46:47.180,0:46:51.549 +there is boundary between the user process +and the top half of the kernel + +0:46:51.549,0:46:54.949 +which is really just like a glorified sovereignty +call + +0:46:54.949,0:46:59.539 +it's a lot like calling into a library routine +like calling strcat, strcpy or something + +0:46:59.539,0:47:00.319 +like that + +0:47:00.319,0:47:03.679 +when you do a system call + +0:47:03.679,0:47:05.650 +we take that same set of parameters + +0:47:05.650,0:47:08.009 +now this is sort of + +0:47:08.009,0:47:09.780 +brick Wall here if you will + +0:47:09.780,0:47:11.380 +that is protecting + +0:47:11.380,0:47:13.680 +the top half of the kernel + +0:47:13.680,0:47:15.299 +from the application + +0:47:15.299,0:47:18.899 +I'll go more into some detail about how that +actually gets implemented + +0:47:18.899,0:47:22.729 +but in essence you can think of it +is is there sort of this whaling Wall and these little + +0:47:22.729,0:47:24.990 +chinks there and you can sort of push a request +through + +0:47:24.990,0:47:28.230 +and somebody other sides sort of pulls that +looks at it and decides whether they're going + +0:47:28.230,0:47:28.690 +to + +0:47:28.690,0:47:30.769 +dain to provide service to you + +0:47:30.769,0:47:34.229 +and if they do then they sort of send it back + +0:47:34.229,0:47:37.649 +well like a library where you can just sort +of reach in and walk around if you want to + +0:47:37.649,0:47:38.290 +you + +0:47:38.290,0:47:40.950 +good programming practices you don't do that +but + +0:47:40.950,0:47:43.049 +you could + +0:47:43.049,0:47:44.579 +all right so + +0:47:44.579,0:47:49.089 +the the top half of the kernel is really looks +a lot like + +0:47:49.089,0:47:50.509 +a big library + +0:47:50.509,0:47:53.509 +%uh it just happens to be a library +routines + +0:47:53.509,0:47:57.599 +that deal with things where processes need +to interact with each other + +0:47:57.599,0:48:01.399 +in fact for many people they don't understand +for what's the difference between the C + +0:48:01.399,0:48:03.259 +library and the top half of the kernel + +0:48:03.259,0:48:08.020 +if it's something that you're doing that +no other process needs to know about + +0:48:08.020,0:48:09.799 +then it can be in the C library + +0:48:09.799,0:48:13.829 +so if you call strcat to concatenate two +strings together + +0:48:13.829,0:48:17.599 +nobody else needs to know you're doing that +you don't need to coordinate with anybody + +0:48:17.599,0:48:19.000 +else that you're doing that + +0:48:19.000,0:48:20.160 +it's just happening + +0:48:20.160,0:48:21.979 +so that goes in the C library. + +0:48:21.979,0:48:24.489 +on the other hand if you're reading or writing +the file + +0:48:24.489,0:48:28.029 +there may be other processes that are also +reading and writing that file + +0:48:28.029,0:48:29.910 +and therefore that + +0:48:29.910,0:48:31.579 +has to be done by the kernel + +0:48:31.579,0:48:33.120 +because they can coordinate + +0:48:33.120,0:48:37.189 +all the different processes that are trying to access +that file. + +0:48:37.189,0:48:40.529 +so the top half of the kernel is pretty straightforward +code + +0:48:40.529,0:48:45.539 +it looks a lot like any other library that +you would write if you look at top half kernel + +0:48:45.539,0:48:49.640 +code you know you see all read, come in +it's got these parameters we Mark around we + +0:48:49.640,0:48:53.719 +get some data that we put it in the buffer and +we return back + +0:48:53.719,0:48:57.470 +and in fact writing code for the top half of +the kernel is + +0:48:57.470,0:48:59.729 +not all that difficult to do + +0:48:59.729,0:49:00.989 +it's + +0:49:00.989,0:49:01.959 +you have + +0:49:01.959,0:49:05.939 +for many of the same properties that you would +when you're writing user level application + +0:49:05.939,0:49:07.529 +code + +0:49:07.529,0:49:11.779 +the bottom half of the kernel is where things +start to get nasty + +0:49:11.779,0:49:14.820 +because the bottom half of the kernel is the part +of the system + +0:49:14.820,0:49:18.769 +that deals with all of the asynchronous events +in the system + +0:49:18.769,0:49:22.179 +is things like device drivers, + +0:49:22.179,0:49:23.779 +timers + +0:49:23.779,0:49:25.010 +that level of thing + +0:49:25.010,0:49:28.029 +that are driven by hardware events + +0:49:28.029,0:49:28.659 +so + +0:49:28.659,0:49:31.459 +for example a packet arrives on the network + +0:49:31.459,0:49:33.670 +that causes an interrupt to come and + +0:49:33.670,0:49:36.729 +that will be handled by the bottom half of +the kernel + +0:49:36.729,0:49:38.829 +and historically + +0:49:38.829,0:49:43.079 +when an interrupt came in it preempted whatever +else was going on + +0:49:43.079,0:49:45.400 +and it ran until it finished and then it returned + +0:49:45.400,0:49:46.539 +and it could not + +0:49:46.539,0:49:49.439 +go to sleep to wait for resources or other +things + +0:49:49.439,0:49:51.339 +%uh in current systems + +0:49:51.339,0:49:54.549 +you can actually go to sleep in the interrupt driver +and waiting for + +0:49:54.549,0:49:56.739 +some other activity to complete + +0:49:56.739,0:49:58.259 +it is however + +0:49:58.259,0:50:00.799 +not a good idea to do that + +0:50:00.799,0:50:01.909 +because + +0:50:01.909,0:50:06.739 +the usual case of most device drivers is they +can finish whatever they're doing in an interrupt + +0:50:06.739,0:50:08.579 +without ever blocking + +0:50:08.579,0:50:09.580 +and so + +0:50:09.580,0:50:13.649 +when an interrupt comes in we assume that you're +not going to sleep + +0:50:13.649,0:50:14.710 +and if you actually + +0:50:14.710,0:50:17.219 +then go to sleep.oh man + +0:50:17.219,0:50:20.469 +you didn't tell us you're going to do this we +have to go off to do a whole lot of other work + +0:50:20.469,0:50:23.029 +that we had originally planned on doing + +0:50:23.029,0:50:25.460 +so if you go to sleep in a device driver + +0:50:25.460,0:50:28.209 +you are taking a very serious performance hit + +0:50:28.209,0:50:31.019 +so it's highly recommended that you don't +do that + +0:50:31.019,0:50:33.130 +but if you have to you can + +0:50:33.130,0:50:35.809 +on it's because of this historic behavior +or + +0:50:35.809,0:50:39.899 +of not being able to sleep in the bottom half +of the kernel + +0:50:39.899,0:50:42.119 +that you have certain properties that have + +0:50:42.119,0:50:44.769 +taken over in device drivers + +0:50:44.769,0:50:45.940 +and that is + +0:50:45.940,0:50:50.369 +that a device driver should be handed all +the resources it needs to get his job done + +0:50:50.369,0:50:54.490 +you don't give a disk device driver +Go read this + +0:50:54.490,0:50:56.549 +and put it somewhere + +0:50:56.549,0:50:57.580 +you have to say + +0:50:57.580,0:50:59.410 +Go read this particular block + +0:50:59.410,0:51:02.650 +here is a chunk of memory that I want that + data to put in + +0:51:02.650,0:51:03.959 +and + +0:51:03.959,0:51:06.169 +notify me when it's done + +0:51:06.169,0:51:06.970 +because + +0:51:06.970,0:51:10.660 +things like allocating memory are classic +places where you end up having to go to sleep + +0:51:10.660,0:51:12.939 +to wait for stuff to happen + +0:51:12.939,0:51:14.449 +and + +0:51:14.449,0:51:16.390 +historically you couldn't do that + +0:51:16.390,0:51:18.640 +even currently don't want to have to do that + +0:51:18.640,0:51:23.400 +so device drivers generally have all +resources pre allocated + +0:51:23.400,0:51:25.169 +and then they can just go + +0:51:25.169,0:51:27.279 +the one place where this doesn't work + +0:51:27.279,0:51:29.029 +is the network + +0:51:29.029,0:51:30.929 +and in particular + +0:51:30.929,0:51:34.630 +you don't know when somebody's going to send +packets to you + +0:51:34.630,0:51:37.040 +you say well you're looking to open connections + +0:51:37.040,0:51:39.360 +but if you're doing something like IP forwarding + +0:51:39.360,0:51:40.969 +there's no + +0:51:40.969,0:51:45.039 +top half state it's dealing with this packets +they're just coming in on one interface being + +0:51:45.039,0:51:46.719 +sent out on another interface + +0:51:46.719,0:51:50.630 +they never pass through any part of the top +half of the kernel + +0:51:50.630,0:51:53.529 +and so in the case of network device drivers + +0:51:53.529,0:51:56.149 +they need to allocate memory + +0:51:56.149,0:51:56.640 +and + +0:51:56.640,0:51:58.829 +if memory gets into short supply + +0:51:58.829,0:52:01.689 +and they try to allocate memory and it's not +available + +0:52:01.689,0:52:05.049 +they historically couldn't wait for memory to be +available + +0:52:05.049,0:52:08.380 +and even in practice today don't wait + + +0:52:08.380,0:52:09.580 +for memory to become available + +0:52:09.580,0:52:12.469 +they simply drop the packet on the floor + +0:52:12.469,0:52:18.109 +it's like well I didn't have any place to +put it sorry oops + +0:52:18.109,0:52:20.940 +now that doesn't cause incorrect behavior + +0:52:20.940,0:52:24.369 +because the higher level protocols will retransmit + +0:52:24.369,0:52:29.140 +but it does cause great performance problems +because retransmission means that connections + +0:52:29.140,0:52:29.879 +stall + +0:52:29.879,0:52:31.110 +they have to back up + +0:52:31.110,0:52:33.010 +they have to resend data + +0:52:33.010,0:52:33.739 +and so on + +0:52:33.739,0:52:38.739 +so you really want to avoid dropping packets +if you can possibly help it + +0:52:38.739,0:52:42.029 +and consequently + +0:52:42.029,0:52:43.420 +we tend to + +0:52:43.420,0:52:46.499 +pre allocate a certain amount of memory for +the network drivers + +0:52:46.499,0:52:48.299 +and + +0:52:48.299,0:52:52.169 +we try very hard to make sure that we're not +going to run out of memory but + +0:52:52.169,0:52:54.869 +if packets come fast enough and we can't deal +with them + +0:52:54.869,0:52:57.940 +as quickly as they are arriving then over short period +of time + +0:52:57.940,0:53:03.489 +we get to the point where we simply have to start +dropping packets + +0:53:03.489,0:53:07.649 +okay this is a part of kernel that you do not wish to +write code for + +0:53:07.649,0:53:10.919 +because it is extremely difficult to +debug + +0:53:10.919,0:53:12.759 +you get these bugs where + +0:53:12.759,0:53:18.779 +the only time it happens is on the third Tuesday +when there's a full moon + +0:53:18.779,0:53:19.300 +and + +0:53:19.300,0:53:24.199 +we have a disk interrupt followed by %uh a +terminal character coming in + +0:53:24.199,0:53:28.289 +and the network packet arriving of size fifteen +twenty two + +0:53:28.289,0:53:30.109 +and when all those things happened + +0:53:30.109,0:53:32.719 +the system panics + +0:53:32.719,0:53:37.380 +and of course there's like it panics +cause you're following some bad pointer + +0:53:37.380,0:53:40.969 +something that should have been there +but was freed some time in the distant past + +0:53:40.969,0:53:42.930 +we are not sure when + +0:53:42.930,0:53:44.049 +and + +0:53:44.049,0:53:47.400 +try to debug things like that is extremely +difficult + +0:53:47.400,0:53:48.509 +and you can + +0:53:48.509,0:53:52.120 +think well I think I found the problem but +it's not reproduceable + +0:53:52.120,0:53:55.530 +you know you have to wait for the next third +Tuesday with a full moon and blah blah blah + +0:53:55.530,0:53:56.950 +to happen + +0:53:56.950,0:53:57.469 +and + +0:53:57.469,0:54:01.449 +you know so you sort of statistically +guess that you fix that you know I was getting + +0:54:01.449,0:54:03.510 +this bug once every three days + +0:54:03.510,0:54:06.099 +and now it's gone for two weeks without happening + +0:54:06.099,0:54:07.239 +did you fix that? + +0:54:07.239,0:54:08.969 +or if you've been lucky + +0:54:08.969,0:54:10.459 +and and it's + +0:54:10.459,0:54:14.349 +that coupled with the fact that you're +dealing with hardware + +0:54:14.349,0:54:18.049 +and hardware rarely works the way it's documented +to work + +0:54:18.049,0:54:21.770 +and so you know they're doing everything that +it says you're supposed to do + +0:54:21.770,0:54:26.260 +it still doesn't work because you didn't set +the fiddle bit over on that other place over + +0:54:26.260,0:54:26.660 +there + +0:54:26.660,0:54:30.479 +that's not documented anywhere but if it's +not said it doesn't work + +0:54:30.479,0:54:33.769 +occasionally + +0:54:33.769,0:54:36.110 +so this is another reason that you really want +of avoid + +0:54:36.110,0:54:40.459 +dealing with this part of the system if +you can possibly help + +0:54:40.459,0:54:44.369 +okay but lets go through and and look at some +of the properties here starting up at + +0:54:44.369,0:54:45.789 +the user process + +0:54:45.789,0:54:47.980 +we're running with + +0:54:47.980,0:54:51.449 +preemptive scheduling + +0:54:51.449,0:54:53.409 +now there's several caveats here + +0:54:53.409,0:54:55.239 +preemptive scheduling is the default + +0:54:55.239,0:54:56.970 +so called shared scheduler + +0:54:56.970,0:55:01.360 +that is what you normally use there are other +schedulers like the real time scheduler + +0:55:01.360,0:55:02.869 +where what I'm saying isn't that true + +0:55:02.869,0:55:05.709 +we'll talk about some of the schedulers was +later + +0:55:05.709,0:55:09.930 +but the usual scheduler that you're running +on under UNIX is a shared scheduler + +0:55:09.930,0:55:13.229 +and under the shared scheduler user applications + +0:55:13.229,0:55:15.159 +run with pre emptive scheduling + +0:55:15.159,0:55:17.449 +and pre emptive scheduling means that + +0:55:17.449,0:55:20.019 +you run at the whim of the system + +0:55:20.019,0:55:21.420 +if it wants you to run + +0:55:21.420,0:55:22.140 +you run + +0:55:22.140,0:55:25.490 +once you to start running you have no guarantee +of how long you're going to run + +0:55:25.490,0:55:29.370 +it might like to run for three instructions +and then decide it doesn't like you many more + +0:55:29.370,0:55:31.150 +it wants to run something else + +0:55:31.150,0:55:35.920 +or you might get to run for several seconds +and in a row with the with no intervening + +0:55:35.920,0:55:37.469 +things interrupting you + +0:55:37.469,0:55:39.719 +you just don't know + +0:55:39.719,0:55:40.969 +and + +0:55:40.969,0:55:42.839 +really all you know is + +0:55:42.839,0:55:43.569 +that + +0:55:43.569,0:55:48.239 +they claim that they're using statistics +and that and that the statistics are fair + +0:55:48.239,0:55:55.059 +and so on average you're going to get a reasonable +amount of time but thats + +0:55:55.059,0:55:57.129 +up to the system you don't control that + +0:55:57.129,0:55:58.439 +the real point here + +0:55:58.439,0:56:01.940 +is that you don't have any way of creating +a critical section + +0:56:01.940,0:56:04.950 +you can't say okay I don't want to be interrupted + +0:56:04.950,0:56:07.429 +during this particular sequence of things + +0:56:07.429,0:56:09.809 +so you have to program + +0:56:09.809,0:56:13.469 +assuming that you may be interrupted at any +point + +0:56:13.469,0:56:14.979 +okay + +0:56:14.979,0:56:18.909 +the next thing is that when you're running +in a user process + +0:56:18.909,0:56:20.719 +you are running in + +0:56:20.719,0:56:24.150 +with the processor in what's called unprivileged +mode + +0:56:24.150,0:56:28.109 +one of the requirements for running any kind +of a UNIX system + +0:56:28.109,0:56:31.759 +is that you have to have a processor that +support privileged and unprivileged + +0:56:31.759,0:56:33.709 +two different modes of operation + +0:56:33.709,0:56:37.049 +in privileged mode which is what the kernel +runs in + +0:56:37.049,0:56:38.950 +the entire repertoire + +0:56:38.950,0:56:40.869 +of the hardware is available + +0:56:40.869,0:56:45.339 +by this I mean you can set all the registers +you can fiddle with the memory management + +0:56:45.339,0:56:47.460 +unit you can initiate I/O + +0:56:47.460,0:56:50.519 +you can access any memory anywhere + +0:56:50.519,0:56:51.919 +etc + +0:56:51.919,0:56:56.540 +when you're running in unprivileged +mode which is what user processes run in and + +0:56:56.540,0:57:00.709 +this a large subset of the instructions which +you cannot execute + +0:57:00.709,0:57:03.480 +you cannot initiate I/O on + +0:57:03.480,0:57:04.209 +devices + +0:57:04.209,0:57:06.770 +you cannot change the memory mapping + +0:57:06.770,0:57:10.209 +you cannot access memory that's not part of +your address space + +0:57:10.209,0:57:13.299 +you cannot execute certain instructions +like halt + +0:57:13.299,0:57:15.589 +and + +0:57:15.589,0:57:19.039 +so in general you are protected + +0:57:19.039,0:57:21.789 +from manipulating anything that's outside of your +address space + +0:57:21.789,0:57:23.759 +this of course is desirable because + +0:57:23.759,0:57:27.059 +when you're running in this unprivileged +mode + +0:57:27.059,0:57:28.300 +you're protected + +0:57:28.300,0:57:31.910 +from other processes manipulating you +and they're protected from you manipulating + +0:57:31.910,0:57:33.079 +them + +0:57:33.079,0:57:36.430 +for those of you that have had that misfortune +to have to use + +0:57:36.430,0:57:39.339 +early versions of windows up to about ninety +eight + +0:57:39.339,0:57:42.470 +they always ran with the processor +running in privileged mode + +0:57:42.470,0:57:44.009 +even in applications + +0:57:44.009,0:57:46.459 +and so either maliciously or accidentally + +0:57:46.459,0:57:50.000 +you could stop on other people address space +or you could stop on the kernel + +0:57:50.000,0:57:53.020 +and a lot of the blue screen of death was +people just + +0:57:53.020,0:57:56.319 +following wild pointers and trashing different +parts of the system + +0:57:56.319,0:57:58.819 +taking everything down + +0:57:58.819,0:58:00.020 +it also makes it + +0:58:00.020,0:58:02.320 +far easier to + +0:58:02.320,0:58:05.459 +implement things like viruses and worms and +other things because + +0:58:05.459,0:58:09.619 +a user application can we rewrite the boot +block on the disk they can just the write down + +0:58:09.619,0:58:13.109 +and manipulate the registers that allow them +to do whatever they want + +0:58:13.109,0:58:16.730 +whereas when you're running in unprivileged +mode you can't write those kinds of + +0:58:16.730,0:58:20.179 +of things + +0:58:20.179,0:58:24.119 +so modern versions of Windows anything from about +2000 on + +0:58:24.119,0:58:26.630 +now run with privileged and unprivileged mode + +0:58:26.630,0:58:28.649 +but UNIX has always required that + +0:58:28.649,0:58:30.219 +and so when you're running an + +0:58:30.219,0:58:31.319 + user process + +0:58:31.319,0:58:33.389 +you cannot block I mean + +0:58:33.389,0:58:37.969 +you cannot execute the instructions which +cause a context switching to occur + +0:58:37.969,0:58:40.349 +you can't pick what's going to run next + +0:58:40.349,0:58:43.140 +you can't make that thing run next all you can +do + +0:58:43.140,0:58:45.189 +is go to the operating system and say + +0:58:45.189,0:58:49.269 +hey I've got nothing to do. pick somebody else +to run + +0:58:49.269,0:58:53.449 +and the operating system is the think they can +then execute the instructions which cause + +0:58:53.449,0:58:57.609 +a different process to be loaded + +0:58:57.609,0:58:59.049 +and run + +0:58:59.049,0:59:03.400 +alright.finally while you're in a user application you're +running on a user stack + +0:59:03.400,0:59:06.410 +that's part of the user's address space + +0:59:06.410,0:59:07.889 +so + +0:59:07.889,0:59:10.819 +part of creating a process gives you a runtime +stack + +0:59:10.819,0:59:14.369 +as part of a virtual address space and so it +can be + +0:59:14.369,0:59:18.199 +more or less up to the limits of the hardware +as big as you want it to be + +0:59:18.199,0:59:19.949 +so if you are running on thirty two-bit processor + +0:59:19.949,0:59:22.819 +you're stack can get the 2 gigabytes + +0:59:22.819,0:59:23.319 +and + +0:59:23.319,0:59:26.839 +the what this means is that anytime you +allocate local variables + +0:59:26.839,0:59:28.529 +you don't have to worry about Oh + +0:59:28.529,0:59:30.609 +is that gonna overrun my stack? + +0:59:30.609,0:59:31.610 +so if you need + +0:59:31.610,0:59:35.519 +a hundred thousand double precision floating +point numbers + +0:59:35.519,0:59:37.189 +you can just as a local variable allocate + +0:59:37.189,0:59:40.269 +an array of size a hundred-thousand type +double + +0:59:40.269,0:59:44.029 +and it just decrements your stack pointer by +hundred hundred thousand bytes + +0:59:44.029,0:59:45.009 +away you go + +0:59:45.009,0:59:47.299 +it's just virtual address space + +0:59:47.299,0:59:49.020 +as you'll see when we get into the kernel + +0:59:49.020,0:59:50.210 +that ceases to be the case |