Kernel components - Linux Tutorial

- You know, the kernel is the heart of Linux system, but what makes it tick? Let's take a look. - [Narrator] You're watching ITProTV. - Hello, welcome back to LPIC-2. I am Zach and this is Don, the Linux king who forgot his crown today, (laughs) he's a humble man, he's humble. So we're going to be looking at the kernel, aren't we? - Absolutely, yeah. - Yeah. - You know deep down inside of Linux is the kernel, it's the core component of it. We know about it, it's there we've even seen it in some of our episodes, but we're going to take a deeper dive here and learn exactly what the kernel is, what it does, where it's located at on the hard drive, how we interact with it, how we extend its functionality, all the various things that we typically just ignore, but as a Linux engineer, you need to know. - So in the past we've described, and we just did, that the kernel is the heart of the Linux system, but in a more technical perspective, what is the kernel really? - All right, so the kernel at the end of the day is a piece of software. It's a piece of software that runs on well, hardware. It is software that is designed specifically to run exactly on top of hardware though. Most software isn't designed like that. Most software doesn't necessarily know what printer you have or what sound card you have, it needs to talk to what's called an abstraction layer. And that abstraction layer is managed by the kernel. The kernel is what's like connecting people together, it's like a middle man. It's connecting the hardware to the software. It's making sure the software is a lot of talking and is allocating it out, It's making sure that multiple pieces of software aren't trying to do the same thing at the same time on the same piece of hardware and wreak havoc. It's also making sure that no piece of software can monopolize the hardware. So it's really acting like a traffic cop sitting in between the programs that you run and the hardware that you access, that's the primary thing that the kernel does. It actually does a handful of other things, it's got many different jobs, but the key component is providing that interface between software and hardware. - So how is the Linux kernel different from other operating systems? - Well you know, every operating system has a kernel, right? Microsoft Windows has the Windows, the NTOS kernel. You've got MacOS, which has the Mac kernel, even though it's built on top of BSD Unix, it actually has a slightly different kernel. And then you've got this, the Linux World, and there's a ton of other operating systems, this would be the longest episode ever if we tried to list them all. But the Linux kernel is basically performing the same function as the NTOS kernel or the Mac kernel and all these other ones. So what makes this one different is that it is completely open source and freely available, and in general, independent commercial interests. We have to think about the origin of the Linux kernel and where it came about. You know, Linux as a whole was a project that came from one person, Linus Torvalds. He was a college student and he was working at university on Unix machines. And when he'd go home at the end of the day, he'd go to his dorm or apartment or whatever. He didn't have access to a Unix machine to a high end, you know, a big mainframe or big high end or whatever. And so he wanted to create an operating system that was basically identical, a clone of Unix, or at least compatible with it so that he could have the same experience at home as he did at work. And that was the origin of the Linux kernel. One person created it, and then he made it freely available so that anybody could provide updates and input to it. And so that's where we are today, where millions of people all across the world are able to just jump in and contribute. And there's over a hundred thousand active developers working on the Linux kernel, as well as the other components that surround the Linux operating system. And that work is an ongoing thing and it's huge. It's gone way beyond one person now, right? I actually looked it up right before the show, as of the filming of this episode. The Linux kernel contains over 27 million lines of code. And Linus didn't write all that, right? A ton of different contributors did. So that's really the big difference. If I want to create some new hardware or new software or some solution, a proprietary thing that I'm going to sell to customers, I don't necessarily need to create an operating system from scratch. I can just go and get the Linux kernel and use that as my start, and then add the tooling around it to make it the fully featured thing that I want. That's really the big thing that differentiates the Linux kernel from other OSS is that, that free availability and open source nature. - So Linux made up of a lot of different pieces. So what parts are considered, I don't know, the part of the kernel? - All right. So, you know, you mentioned the different pieces, right? The kernel by itself is missing a ton of stuff, right? So the kernel, it's almost easier to talk about what it doesn't do than what it does do. So for example, the kernel does not provide us with a graphical user interface. The kernel does not give us any really human interactivity at all. It doesn't give us the bash shell that we're used to, you know, that doesn't come right. The services we run like Apache web services, even our knit systems like system D and CISV in it, that's not the kernel, right? The kernel runs before all of those things, the kernel makes all of those things possible, but it doesn't actually provide those things. So you'll hear some people like Richard Stallman. He's really big on this, where if you call Linux an operating system, he freaks out, like, "It's not an operating system." Because technically Linux is just the kernel. And when you surround it with tools like the GNU tool set, now you have the bash shell and the VI editor and maybe X windows, and the other things that kind of go along with it to build out a full operating system. And that's why with Linux, you get what are called Linux distros or distributions. They're distributing the kernel with a bunch of other things added onto it. So the kernel itself though, what it's providing is a memory manager. So it's controlling things going in and out of memory. It's got a process manager. So when an application executes, it's actually the kernel it's executing that application and passing that into memory. It has hardware control, obviously it's initializing all of the hardware and providing the driver interface to it. So it's what's actually talking to the hardware components in your computer. And it's got our disc file system. So when you have a file system that's formatted with EXTF EXT four, something like that, right? In that scenario, it's the kernel that's speaking that language to reach out and write to that hardware. And there's other driver systems that are tacked onto it, but the kernel is kind of at the heart and soul of it. So it's managing the various components of our system. That's what the kernel does. Doesn't give us human interaction. And in a lot of cases it doesn't even give software interaction. There's still some intermediary in between the kernel and those applications, but that's basically what it's doing. And then we surround it with tools. So you could have two systems like, Zach you could be running, you know, Linux on your laptop and I could be running Linux on my laptop, but they could be completely different in the way they looked and functioned. You could have a different shell, a different DUI, a whole different set of tools, but still be running that kernel. So that's kind of how that breaks down. - Is the kernel file we can see in the hard drive or is it stored in some other special way? - It is just a file on the hard drive, yeah. So, you know, I mentioned 27 million lines of code. We don't normally see that code. It's compiled into a binary. - Is it real? Is it real. Do you have a board? - No, you know, I can guarantee you that even Linus Torvalds has not read all 27.8 million lines of code, right? It's just too much. That's why there's a team of people that work on this stuff. But for us, for the engineers, people in the field, the binary, I mean, the kernel is a binary. So it's a program, it's been compiled and it's already sitting there on our hard drive. Now it might be compressed, it might not, but we can actually spot it. Here, I'll pull up my machine. I'm running Ubuntu 20.04, at least I think I am. Let me just verify that real quick. Yeah. So I'm on Ubuntu 20.04. And if I just go and take a look at my slash boot folder, slash boot. Slash boot's a weird folder, right? If you've got Luxe encryption turned on, slash boot is not encrypted. Because when your system turns on, it needs to be able to load the kernel. And so the kernel is stored in a non-encrypted petition, which is the slash boot. It has various protections on it as well. But when we take a look in there, we'll actually see a bunch of different files. All right? Some of these files may be meaningful to us, many of them aren't. But the key one that I want to highlight is this guy right here, vmLinuz with a Z, which is our kernel. That's the actual kernel right there, that little file. Now it's SIM linked. I can see it's a light blue in color. So let me do an LS dash L so we can see a longer output, actually, let me an L H so we get human readable sizes. And I can see that vmLinuz with a Z is SIM linked to 0.2 vmLinuz, 5.8.0 dash 45 dash generic, which means I'm currently running the Linux 5.8.0 kernel. That's what I'm running. And as of the filming of this episode right now, right, checked last night actually, 5.9.19 is the current launch. I'm a little bit behind that's common, right? It is rare, super rare. In fact, almost never that a Linux show is running the version of the kernel. The various providers need time to test it, try it out, make sure everything works, update software to be compatible and that takes time. So it's not unusual to be behind one version or even several. I bet if I were to fire up a red hat machine right now, I'm probably running kernel 2.6. I don't think they've well, actually they might have made the jump to four. They definitely haven't made the jump to five, at least as of the filming of this episode. So don't expect to necessarily see the bleeding edge of technology on this one, but that's the kernel right there. It is compressed. You'll see that mine is well, the SIM link is only 24 bytes. That's not very spectacular, but if we find the vmLinuz 5.8.0 dash 45, that mine is sitting at about, oh yes, that's 48. Here we go 45, 9.4 megabytes in size. And actually this is worth pointing out. Notice how I've got the 5.80 dash 45. I've also got 5.4.0, it's a previous version. So I've done an update and it's kept the previous version laying around in case we need to refer to it. But that is the actual binary file right there with the version number 9.5 megabytes attached to it. Now, you might not see this on your system. If you look, you might have one that's called vmLinux with an X, that makes a little more sense. Doesn't it? Well, the Z just means that mine is compressed with G zip. And if you see it with an X it's uncompressed right? Not every distro compresses the kernel, it has to decompress it when it loads it into memory anyway, but some of them like to save space. And so they compress it with a Z. You might also see it with a slightly different name, like Z image or BZ image, which would indicate if it was compressed with G zip or B zip, which is another compression protocol. So you may see a variation in that name, but the most common one, the one you're going to see in Rel, Santos, Fedora, Ubuntu, Dian, Mint, right? All the big distros is going to be this vmLinuz, just like it is right there. - I've heard the Linux kernel described as monolithic. Now, does that mean the entire OS is basically the kernel file? - So Linux does have a monolithic kernel. If you're not familiar with that term, kernels are dropped into two classes. They're either monolithic or micro kernels. A monolithic kernel means that really everything that's part of the operating system runs in kernel space as part of kernel memory and it's kind of protected area. And so everything should be technically in the kernel, right? But if that was the case, the kernel would be huge, not what I'm seeing here, which is, you know, 9.4 megabytes. That's not very big at all, right? If everything was in there, it would be hundreds of megabytes in size, maybe even larger, just depending. So it is monolithic, but it kind of does something interesting. The alternative would be to be a micro kernel. A micro kernel is where the kernel is as small as possible, and everything else runs in user space. So in a different set of memory, and then is able to call the kernel through some safe API, through some other mechanism. Micro kernels are a lot more complex, a lot more dependencies and things of that nature. A monolithic kernel is typically faster because everything's right there in the kernel. And so what Linux does is something interesting. So it is monolithic, right? But it leverages what are called kernel modules. Modules can be dynamically loaded. Think of a module, as an example, let's say it's a driver, right? Maybe I've got a raid control, like a Hewlett Packard raid controller inside of my server. Well, I need a driver for that. I need a firmware file that's be able to operate with that raid controller. And if I've got a monolithic kernel, then that driver would need to be part of the kernel. All right, fine. So now my system boots, it's got access to it and there we go. We can see the raid controller wonderful, right? But if I take that same raid controller or that same kernel, and I run it on my laptop, my laptop doesn't have that raid controller. So now I've loaded some kernel data that I don't need. It's just wasting space and not really achieving anything. So with kernel modules, it allows us to dynamically load them if we need them and dynamically unload them, if we don't need them. And so the kernel can reach out and grab them. So on the file system, no, not everything is in the kernel. It's all kind of separate spread across the file system. But when it's loaded it into RAM, the kernel is then able to reach out, grab the modules it needs and load them into kernel space. And so in RAM, yes, it is all part. It's monolithic, one giant memory address space all together with all the various drivers and firmware and other things that we need to make and operate in one big thing. But that only exists in RAM. And when you turn your computer off, it all goes away. poof, right? And when you power back on, it starts to load that stuff back up again, right? Now, some of those modules are needed before the kernel is even able to start. I mentioned raid controllers, raid controllers are an example of that. And so you'll notice some other files here, let me highlight this one, initrd.img. And you'll see where it's got the same version number as my kernel, that stands for initialization RAM disc image. It's a RAM disc, it's a virtual disc. When you boot up the booter or grub in this case, grub will look for the kernel and it will look for the RAM disc at the same time. It'll read that disc into RAM and create a virtual temporary file system. And that will contain the modules that the kernel immediately needs. It'll be able to use those to then read the disc and be able to load everything else once it's able to read the disc, it usually comes down to storage. There are a few other types of drivers that might be in there. So your initrd file may vary widely in size, depending on what you've got. Mine is currently 51 megabytes. Yours could be larger, could be smaller, just depends on your hardware. And I do want to show you where the modules are stored on the hard drive, the rest of the modules. If we browse to slash lib, slash modules, that's the folder that actually contains all the modules and you'll see different folders for each version of your kernel. So I'm currently running 5.8. And now when we take a look in here, we'll see something kind of interesting. Let me zoom in on this a bit. You'll see, I've got two folders. One is initrd, and one is kernel. So modules that are in the initrd folder, those are the ones that are going to end up in that RAM disc, and then ones that are in the kernel folder. Those are the ones that are loaded afterwards, after the kernel's already started and accessing the file system. And we can browse around. There's not really much need to browse around in there, but if you ever get bored, you can jump in and you'll see folders for all of the different subsystems. So, you know, whatever you can think of has hardware, you'll see folders for it here. Like networks, sound, drivers, crypto, block storage, you name it, they've got folders. So for example, if I were to dig into network, let's go into net. I'll start to see all sorts of network names from vendors and protocols and technologies that we, you know, may or may not be familiar with, but you'll see a lot of stuff in here are based on whatever it is that we're trying to do. So kind of neat, to be able to see this and understand where these models are and the kernel has access to all of this. So if I. - So if I want to learn more about the kernel and how it works and who wouldn't. Where should I look? - You know, there's a few ways to learn about it. If you have the kernel source code installed, then you actually have the kernel's documentation on your disc. You can check for that. Odds are you don't have it, but you can always check by taking a look at slash USR, slash SRC. And when you look in there, if you've got a folder, that's the Linux source, it'll be called Linux source, or have a similar name with your kernel version number on it. If you see that, then you've got the source code and you can change into that folder. And you'll see a folder called documentation right there, and you can read through it and so on. I will tell you I never do that. And the reason is it's really hard to read, the formatting for that documentation is a string, they actually use a system. I forget the name. I think I've got it written down somewhere around here, but it's this weird file format. That just, it's text but it's hard to read. So typically if I want to learn more about the kernel, I use the online documentation. If you go to kernel.org, kernel.org is a website that is set up specifically for this, for you to be able to get the Linux kernel source code, as well as the documentation. And this is the official project from Linux to vault. So this isn't like a third party, one off type thing. And so I can come in and I can see information about the kernel, the kernel build system, and all sorts of other documentation to learn about it, to figure out, you know, how the kernel does various things, what modules are available and really learn. Be careful, all of this documentation is specifically about the kernel. So it's not distro specific. So it doesn't think you're running Ubuntu or Rel, it just thinks you're running the Linux kernel. So just be aware the documentation is written that way and we have to be a little bit careful with that. - So I noticed Linux headers in your folder list. - Oh yeah. Yeah. - So what are those? - So the headers, so when I breezed over this, the slash USR, slash SRC folder, I have a bunch that are called Linux dash headers, not Linux dash source, right? So this is just, well, they're just headers. Headers are an interesting thing. So anytime you try and compile like a binary, right? The binary has to be able to talk to the Linux API and talk to the kernel. So it has to have various components, it's got to pass various tests in order to compile. Well, in theory, if you had the full Linux source code, you could run whatever application you were compiling against the source code and make sure it passed its checks. But what they did is they said people don't want to have the entire set of source code on their disc. So they created a subset called headers. And what the headers has is really a list of all the functions that are defined in the Linux source code and the output they expect. And so it can run a compilation test against your application and see what output it generates and see if it matches the output that the functions expect is input. And as long as they match, that's good, it passed and you don't need to run it against the entire source code. So it makes compiling things faster. It takes less space on the hard drive, and that's what the headers are. I mentioned, you probably don't have the Linux source code on your disc. You probably do have the Linux headers. It's not a guarantee. If you've never compiled anything, then you probably don't have the headers. But if you've compiled anything, then the odds are, you've got the Linux headers already on your disc, and that's what those folders are. So don't confuse those with source code, a lot of people do. - Fantastic information. Thank you, Don. Thank you for joining us for this episode of LPIC-2, before we move on though, anything else from your majesty. - All right. So in this episode, we were really learning. We were exploring the kernel, what it does, where it's on the hard drive and so on. We didn't actually do anything, it's kind of boring. So stay tuned 'cause in our next two episodes we are going to mess with our kernel modules in the next episode. And then we'll compile a kernel in the episode after that. So be sure to check those out. - More kernels of kernels coming up, I'm guessing on LPIC-2. Make sure you watch every episode, you'll be glad you did. Thanks for watching. - [Narrator] Thank you for watching ITproTv.