Development

Developer communityedit

The community of Linux kernel developers comprises about 5000-6000 members. According to the "2017 State of Linux Kernel Development", a study issued by the Linux Foundation, covering the commits for the releases 4.8 to 4.13, about 1500 developers were contributing from about 200-250 companies on average. The top 30 developers contributed a little more than 16% of the code. As of companies, the top contributors are Intel (13.1%) and Red Hat (7.2%), Linaro (5.6%), IBM (4.1%), the second and fifth places are held by the 'none' (8.2%) and 'unknown' (4.1%) categories.

Instead of a roadmap, there are technical guidelines. Instead of a central resource allocation, there are persons and companies who all have a stake in the further development of the Linux kernel, quite independently from one another: People like Linus Torvalds and I don’t plan the kernel evolution. We don’t sit there and think up the roadmap for the next two years, then assign resources to the various new features. That's because we don’t have any resources. The resources are all owned by the various corporations who use and contribute to Linux, as well as by the various independent contributors out there. It's those people who own the resources who decide...

— Andrew Morton, 2005

Source code managementedit

The Linux development community uses Git to manage the source code. Git users clone the latest version of Torvalds' tree with git-clone(1) and keep it up to date using git-pull(1). Contributions are submitted as patches, in the form of text messages on the LKML (and often also on other mailing lists dedicated to particular subsystems). The patches must conform to a set of rules and to a formal language that, among other things, describes which lines of code are to be deleted and what others are to be added to the specified files. These patches can be automatically processed so that system administrators can apply them in order to make just some changes to the code or to incrementally upgrade to the next version. Linux is distributed also in GNU zip (gzip) and bzip2 formats.

Submitting code to the kerneledit

A developer who wants to change the Linux kernel starts with developing and testing that change. Depending on how significant the change is and how many subsystems it modifies, the change will either be submitted as a single patch or in multiple patches of source code. In case of a single subsystem that is maintained by a single maintainer, these patches are sent as e-mails to the maintainer of the subsystem with the appropriate mailing list in Cc. The maintainer and the readers of the mailing list will review the patches and provide feedback. Once the review process has finished the subsystem maintainer accepts the patches in the relevant Git kernel tree. If the changes to the Linux kernel are bug fixes that are considered important enough, a pull request for the patches will be sent to Torvalds within a few days. Otherwise, a pull request will be sent to Torvalds during the next merge window. The merge window usually lasts two weeks and starts immediately after the release of the previous kernel version. The Git kernel source tree names all developers who have contributed to the Linux kernel in the Credits directory and all subsystem maintainers are listed in Maintainers.

Programming language and coding styleedit

Linux is written in a special C programming language supported by GCC, a compiler that extends in many ways the C standard, for example using inline sections of code written in the assembly language (in GCC's "AT&T-style" syntax) of the target architecture. Since 2002 all the code must adhere to the 21 rules comprising the Linux Kernel Coding Style.

GNU toolchainedit

The GNU Compiler Collection (GCC or GNU cc) is the default compiler for the mainline Linux sources and it is invoked by a utility called make. Then, the GNU Assembler (more often called GAS or GNU as) outputs the object files from the GCC generated assembly code. Finally, the GNU Linker (GNU ld) is used to produce a statically linked executable kernel file called vmlinux. as and ld are part of a package called GNU binutils. The above-mentioned tools are collectively known as the GNU toolchain.

Compiler compatibilityedit

GCC was for a long time the only compiler capable of correctly building Linux. In 2004, Intel claimed to have modified the kernel so that its C compiler was also capable of compiling it. There was another such reported success in 2009, with a modified 2.6.22 version.

Since 2010, effort has been underway to build Linux with Clang, an alternative compiler for the C language; as of 12 April 2014, the official kernel could almost be compiled by Clang. The project dedicated to this effort is named LLVMLinux after the LLVM compiler infrastructure upon which Clang is built. LLVMLinux does not aim to fork either Linux or the LLVM, therefore it is a meta-project composed of patches that are eventually submitted to the upstream projects. By enabling Linux to be compiled by Clang, developers may benefit from shorter compilation times.

In 2017, developers completed upstreaming patches to support building the Linux kernel with Clang in the 4.15 release, having backported support for X86-64 and AArch64 to the 4.4, 4.9, and 4.14 branches of the stable kernel tree. Google's Pixel 2 shipped with the first Clang built Linux kernel, though patches for Pixel (1st generation) did exist. 2018 saw ChromeOS move to building kernels with Clang by default, while Android (operating system) made Clang and LLVM's linker LLD required for kernel builds in 2019. Google moved its production kernel used throughout its datacenters to being built with Clang in 2020. Today, the ClangBuiltLinux group coordinates fixes to both Linux and LLVM to ensure compatibility, both composed of members from LLVMLinux and having upstreamed patches from LLVMLinux.

Kernel debuggingedit

Linux bugs can be very hard to find out and fix, firstly because of kernel interaction with userspace and hardware and secondly because they might be caused from a wider number of reasons than those which can affect user programs. A few examples of the underlying causes are semantic errors in code, misuse of synchronization primitives, and an incorrect hardware management.

A report of a non-fatal bug in the kernel is called an "oops"; such deviations from correct behavior of the Linux kernel may allow continued operation with compromised reliability.

A critical and fatal error is reported via the "panic()" function. It prints a message and then halts the kernel.

One of the most common techniques used to find out bugs in code is debugging by printing. For this purpose Linux provides an in-kernel API called printk() which stores messages in a circular buffer. The syslog(2) system call is used for reading and/or clearing the kernel message ring buffer and for setting the maximum log level of the messages to be sent to the console (i.e., one of the eight KERN_* parameters of printk(), which tell the severity of the condition reported); usually it is invoked via the glibC wrapper klogctl(3). Kernel messages are also exported to userland through the /dev/kmsg interface (e.g., systemd-journald reads that interface and by default append the messages to /var/log/journal).

Another fundamental technique for debugging a running kernel is tracing. The ftrace mechanism is a Linux internal tracer; it is used for monitoring and debugging Linux at runtime and it can also analyze user space latencies due to kernel misbehavior. Furthermore, ftrace allows users to trace Linux at boot-time.

kprobes and kretprobes can break (like debuggers in userspace) into Linux and non-disruptively collect information. kprobes can be inserted into code at (almost) any address, while kretprobes work at function return. uprobes have similar purposes but they also have some differences in usage and implementation.

With KGDB Linux can be debugged in much the same way as userspace programs. KGDB requires an additional machine that runs GDB and that is connected to the target to be debugged using a serial cable or Ethernet.

Development modeledit

The Linux kernel project integrates new code on a rolling basis. Software checked into the project must work and compile without error. For each kernel subsystem there is a maintainer who is responsible for reviewing patches against the kernel code standards and keeps a queue of patches that can be submitted to Linus Torvalds within a merge window of several weeks. Patches are merged by Torvalds into the source code of the prior stable Linux kernel release, creating the -rc release candidate for the next stable kernel. Once the merge window is closed only fixes to the new code in the development release are accepted. The -rc development release of the kernel goes through regression tests and once it is judged to be stable by Torvalds and the kernel subsystem maintainers a new Linux kernel is released and the development process starts all over again.

Developers who feel treated unfairly can report this to the Linux Foundation's Technical Advisory Board. In July 2013 the maintainer of the USB 3.0 driver Sarah Sharp asked Torvalds to address the abusive commentary in the kernel development community. In 2014, Sharp backed out of Linux kernel development, saying that "The focus on technical excellence, in combination with overloaded maintainers, and people with different cultural and social norms, means that Linux kernel maintainers are often blunt, rude, or brutal to get their job done". At the linux.conf.au (LCA) conference in 2018, developers expressed the view that the culture of the community has gotten much better in the past few years. Daniel Vetter, the maintainer of the Intel drm/i915 graphics kernel driver, commented that the "rather violent language and discussion" in the kernel community has decreased or disappeared.

Laurent Pinchart asked developers for feedback on their experience with the kernel community at the 2017 Embedded Linux Conference Europe. The issues brought up were a few days later discussed at the Maintainers Summit. Concerns over the lack of consistency in how maintainers responded to patches submitted by developers were echoed by Shuah Khan, the maintainer of the kernel self-test framework. Torvalds contended that there would never be consistency in the handling of patches because different kernel subsystems have over time adopted different development processes. Therefore, it was agreed upon that each kernel subsystem maintainer would document the rules for patch acceptance.

Linux forksedit

There are certain communities that develop kernels based on the official Linux. Some interesting bits of code from these forks (i.e., a slang term meaning "derived projects") that include Linux-libre, Compute Node Linux, INK, L4Linux, RTLinux, and User-Mode Linux (UML) have been merged into the mainline. Some operating systems developed for mobile phones initially used heavily modified versions of Linux, including Google Android, Firefox OS, HP webOS, Nokia Maemo and Jolla Sailfish OS. In 2010, the Linux community criticised Google for effectively starting its own kernel tree:

This means that any drivers written for Android hardware platforms, can not get merged into the main kernel tree because they have dependencies on code that only lives in Google's kernel tree, causing it to fail to build in the kernel.org tree. Because of this, Google has now prevented a large chunk of hardware drivers and platform code from ever getting merged into the main kernel tree. Effectively creating a kernel branch that a number of different vendors are now relying on.

— Greg Kroah-Hartman, 2010

Today Android uses a slightly customized Linux where changes are implemented in device drivers so that little or no change to the core kernel code is required. Android developers also submit patches to the official Linux that finally can boot the Android operating system. For example, a Nexus 7 can boot and run the mainline Linux.

At a 2001 presentation at the Computer History Museum, Linus Torvalds had this to say in response to a question about distributions of Linux using precisely the same kernel sources or not:

They're not...well they are, and they're not. There is no single kernel. Every single distribution has their own changes. That's been going on since pretty much day one. I don't know if you may remember Yggdrasil was known for having quite extreme changes to the kernel and even today all of the major vendors have their own tweaks because they have some portion of the market they're interested in and quite frankly that's how it should be. Because if everybody expects one person, me, to be able to track everything that's not the point of GPL. That's not the point of having an open system. So actually the fact that a distribution decides that something is so important to them that they will add patches for even when it's not in the standard kernel, that's a really good sign for me. So that's for example how something like ReiserFS got added. And the reason why ReiserFS is the first journaling filesystem that was integrated in the standard kernel was not because I love Hans Reiser. It was because SUSE actually started shipping with ReiserFS as their standard kernel, which told me "ok." This is actually in production use. Normal People are doing this. They must know something I don't know. So in a very real sense what a lot of distribution houses do, they are part of this "let's make our own branch" and "let's make our changes to this." And because of the GPL, I can take the best portions of them.

— Linus Torvalds, 2001

Development community conflictsedit

There have been several notable conflicts among Linux kernel developers. Examples of such conflicts are:

• In July 2007, Con Kolivas announced that he would cease developing for the Linux kernel.
• In July 2009, Alan Cox quit his role as the TTY layer maintainer after disagreement with Linus Torvalds.
• In December 2010, there was a discussion between Linux SCSI maintainer James Bottomley and SCST maintainer Vladislav Bolkhovitin about which SCSI target stack should be included in the Linux kernel. This made some Linux users upset.
• In June 2012, Torvalds made it very clear that he did not agree with NVIDIA releasing its drivers as closed.
• In April 2014, Torvalds banned Kay Sievers from submitting patches to the Linux kernel for failing to deal with bugs that caused systemd to negatively interact with the kernel.
• In October 2014, Lennart Poettering accused Torvalds of tolerating the rough discussion style on Linux kernel related mailing lists and of being a bad role model.
• In March 2015, Christoph Hellwig filed a lawsuit against VMware for infringement of the copyright on the Linux kernel. Linus Torvalds made it clear that he did not agree with this and similar initiatives by calling lawyers a festering disease.

Prominent Linux kernel developers have been aware of the importance of avoiding conflicts between developers. For a long time there was no code of conduct for kernel developers due to opposition by Linus Torvalds. However, a Linux Kernel Code of Conflict was introduced on 8 March 2015. It was replaced on 16 September 2018 by a new Code of Conduct based on the Contributor Covenant. This coincided with a public apology by Torvalds and a brief break from kernel development. On 30 November 2018, complying with the Code of Conduct, Jarkko Sakkinen of Intel sent out patches replacing instances of "fuck" appearing in source code comments with suitable versions focused on the word 'hug'.

Codebaseedit

As of 2020update, the 5.9 release of the Linux kernel had around 27.8 million lines of code, roughly 14% of the code is part of the "core" (arch, kernel and mm directories) while 60% is drivers.

Linux is evolution, not intelligent design!

— Linus Torvalds, 2005

Estimated cost to redevelopedit

The cost to redevelop the Linux kernel version 2.6.0 in a traditional proprietary development setting has been estimated to be US$612 million (€467M, £394M) in 2004 prices using the COCOMO person-month estimation model. In 2006, a study funded by the European Union put the redevelopment cost of kernel version 2.6.8 higher, at €882M ($1.14bn, £744M).

This topic was revisited in October 2008 by Amanda McPherson, Brian Proffitt, and Ron Hale-Evans. Using David A. Wheeler's methodology, they estimated redevelopment of the 2.6.25 kernel now costs $1.3bn (part of a total $10.8bn to redevelop Fedora 9). Again, Garcia-Garcia and Alonso de Magdaleno from University of Oviedo (Spain) estimate that the value annually added to kernel was about €100M between 2005 and 2007 and €225M in 2008, it would cost also more than €1bn (about $1.4bn as of February 2010) to develop in the European Union.

As of 7 March 2011update, using then-current LOC (lines of code) of a 2.6.x Linux kernel and wage numbers with David A. Wheeler's calculations it would cost approximately $3bn (about €2.2bn) to redevelop the Linux kernel as it keeps getting bigger. An updated calculation as of 26 September 2018update, using then-current 20,088,609 LOC (lines of code) for the 4.14.14 Linux kernel and the current US National average programmer salary of $75,506 show it would cost approximately $14,725,449,000 dollars (£11,191,341,000) to rewrite the existing code.

Maintenance and long-term supportedit

The latest kernel version and older kernel versions are maintained separately. Most latest kernel releases were supervised by Linus Torvalds. Current versions are released by Greg Kroah-Hartman.

The Linux kernel developer community maintains a stable kernel by applying fixes for software bugs that have been discovered during the development of the subsequent stable kernel. Therefore, www.kernel.org will always list two stable kernels. The next stable Linux kernel is now released only 8 to 12 weeks later. Therefore, the Linux kernel maintainers have designated some stable kernel releases as longterm, these long-term support Linux kernels are updated with bug fixes for two or more years. In November 2019 there were five longterm Linux kernels: 4.19.84, 4.14.154, 4.9.201, 4.4.201 and 3.16.76. The full list of releases is at Linux kernel version history.

Relation with Linux distributionsedit

Most Linux users run a kernel supplied by their Linux distribution. Some distributions ship the "vanilla" or "stable" kernels. However, several Linux distribution vendors (such as Red Hat and Debian) maintain another set of Linux kernel branches which are integrated into their products. These are usually updated at a slower pace compared to the "vanilla" branch, and they usually include all fixes from the relevant "stable" branch, but at the same time they can also add support for drivers or features which had not been released in the "vanilla" version the distribution vendor started basing their branch from.