Day 17 - Build from the source

Complementary video

INTRO

A few days ago we saw how to authorise extra repositories for apt-cache to search when we need unusual applications, or perhaps more recent versions than those in the standard repositories.

Today we’re going one step further - literally going to “go to the source”. This is not something to be done lightly - the whole reason for package managers is to make your life easy - but occasionally it is justified, and it is something you need to be aware of and comfortable with.

The applications we’ve been installing up to this point have come from repositories. The files there are “binaries” - pre-compiled, and often customised by your distro. What might not be clear is that your distro gets these applications from a diverse range of un-coordinated development projects (the “upstream”), and these developers are continuously working on new versions. We’ll go to one of these, download the source, compile and install it.

(Another big part of what package managers like apt do, is to identify and install any required “dependencies”. In the Linux world many open source apps take advantage of existing infrastructure in this way, but it can be a very tricky thing to resolve manually. However, the app we’re installing today from source is relatively unusual in being completly standalone).

YOUR TASKS TODAY

Download a source code tarball
Extract and build the source

FIRST WE NEED THE ESSENTIALS

Projects normally provide their applications as “source files”, written in the C, C++ or other computer languages. We’re going to pull down such a source file, but it won’t be any use to us until we compile it into an “executable” - a program that our server can execute. So, we’ll need to first install a standard bundle of common compilers and similar tools. On Ubuntu, the package of such tools is called “build-essential”. Install it like this:

sudo apt install build-essential

GETTING THE SOURCE

First, test that you already have nmap installed, and type nmap -V to see what version you have. This is the version installed from your standard repositories. Next, type: which nmap - to see where the executable is stored.

Now let’s go to the “Project Page” for the developers http://nmap.org/ and grab the very latest cutting-edge version. Look for the download page, then the section “Source Code Distribution” and the link for the “Latest development nmap release tarball” and note the URL for it - something like:

 https://nmap.org/dist/nmap-7.70.tar.bz2

This is version 7.70, the latest development release when these notes were written, but it may be different now. So now we’ll pull this down to your server. The first question is where to put it - we’ll put it in your home directory, so change to your home directory with:

cd

then simply using wget (“web get”), to download the file like this:

wget -v https://nmap.org/dist/nmap-7.70.tar.bz2

The -v (for verbose), gives some feedback so that you can see what’s happening. Once it’s finished, check by listing your directory contents:

ls -ltr

As we’ve learnt, the end of the filename is typically a clue to the file’s format - in this case “.bz2” signals that it’s a tarball compressed with the bz2 algorithm. While we could uncompress this then un-combine the files in two steps, it can be done with one command - like this:

tar -j -x -v -f nmap-7.70.tar.bz2

....where the -j means “uncompress a bz2 file first”, -x is extract, -v is verbose - and -f says “the filename comes next”. Normally we’d actually do this more concisely as:

tar -jxvf nmap-7.70.tar.bz2

So, lets see the results,

ls -ltr

Remembering that directories have a leading “d” in the listing, you’ll see that a directory has been created :

 -rw-r--r--  1 steve  steve  21633731    2011-10-01 06:46 nmap-7.70.tar.bz2
 drwxr-xr-x 20 steve  steve  4096        2011-10-01 06:06 nmap-7.70

Now explore the contents of this with mc or simply cd nmap-7.70 - you should be able to use ls and less find and read the actual source code. Even if you know no programming, the comments can be entertaining reading.

By convention, source files will typically include in their root directory a series of text files in uppercase such as: README and INSTALLATION. Look for these, and read them using more or less. It’s important to realise that the programmers of the “upstream” project are not writing for Ubuntu, CentOS - or even Linux. They have written a correct working program in C or C++ etc and made it available, but it’s up to us to figure out how to compile it for our operating system, chip type etc. (This hopefully gives a little insight into the value that distributions such as CentOS, Ubuntu and utilities such as apt, yum etc add, and how tough it would be to create your own Linux From Scratch)

So, in this case we see an INSTALL file that says something terse like:

 Ideally, you should be able to just type:

 ./configure
 make
 make install

 For far more in-depth compilation, installation, and removal notes
 read the Nmap Install Guide at http://nmap.org/install/ .

In fact, this is fairly standard for many packages. Here’s what each of the steps does:

./configure - is a script which checks your server (ie to see whether it’s ARM or Intel based, 32 or 64-bit, which compiler you have etc). It can also be given parameters to tailor the compilation of the software, such as to not include any extra support for running in a GUI environment - something that would make sense on a “headless” (remote text-only server), or to optimize for minimum memory use at the expense of speed - as might make sense if your server has very little RAM. If asked any questions, just take the defaults - and don’t panic if you get some WARNING messages, chances are that all will be well.
make - compiles the software, typically calling the GNU compiler gcc. This may generate lots of scary looking text, and take a minute or two - or as much as an hour or two for very large packages like LibreOffice.
make install - this step takes the compiled files, and installs that plus documentation to your system and in some cases will setup services and scheduled tasks etc. Until now you’ve just been working in your home directory, but this step installs to the system for all users, so requires root privileges. Because of this, you’ll need to actually run: sudo make install. If asked any questions, just take the defaults.

Now, potentially this last step will have overwritten the nmap you already had, but more likely this new one has been installed into a different place.

In general /bin is for key parts of the operating system, /usr/bin for less critical utilities and /usr/local/bin for software you’ve chosen to manually install yourself. When you type a command it will search through each of the directories given in your PATH environment variable, and start the first match. So, if /bin/nmap exists, it will run instead of /usr/local/bin - but if you give the “full path” to the version you want - such as /usr/local/bin/nmap - it will run that version instead.

The “locate” command allows very fast searching for files, but because these files have only just been added, we’ll need to manually update the index of files:

sudo updatedb

Then to search the index:

locate bin/nmap

This should find both your old and new copies of nmap

Now try running each, for example:

/usr/bin/nmap -V

/usr/local/bin/nmap -V

The nmap utility relies on no other package or library, so it is very easy to install from source. Most other packages have many “dependencies”, so installing them from source by hand can be pretty challenging even when well explained (look at: http://oss.oetiker.ch/smokeping/doc/smokeping_install.en.html for a good example).

NOTE: Because you’ve done all this outside of the apt system, this binary won’t get updates when you run apt update. Not a big issue with a utility like nmap probably, but for anything that runs as an exposed service it’s important that you understand that you now have to track security alerts for the application (and all of its dependencies), and install the later fixed versions when they’re available. This is a significant pain/risk for a production server.

POSTING YOUR PROGRESS

Pat yourself on the back if you succeeded today - and let us know in the forum.

EXTENSION

Research some distributions where “from source” is normal:

None of these is typically used in production servers, but investigating any of them will certainly increase your knowledge of how Linux works “under the covers” - asking you to make many choices that the production-ready distros such as RHEL and Ubuntu do on your behalf by choosing what they see as sensible defaults.

RESOURCES

Some rights reserved. Check the license terms here