self hosting on the Pi3

On a sunny day (Sun, 1 Mar 2020 03:28:59 +0000 (UTC)) it happened bob prohaska wrote in :

Yes why not, google finds this for the kernel:

have not tried to compile it though!

The challenge is to find the complete set of sources that corresponds to what you have installed. I've never succeeded in doing that with any Linux installation - to be fair I've not tried in many years. With the BSDs OTOH it's easy because they're set up that way. I have no idea about self-hosting RiscOS.

--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
The computer obeys and wins.                |    licences available see 
You lose and Bill collects.                 |    http://www.sohara.org/

Should be - source is available as part of the Gnu license.

-- Those who want slavery should have the grace to name it by its proper name. They must face the full meaning of that which they are advocating or condoning; the full, exact, specific meaning of collectivism, of its logical implications, of the principles upon which it is based, and of the ultimate consequences to which these principles will lead. They must face it, then decide whether this is what they want or not.

Ayn Rand.

On a sunny day (Sun, 1 Mar 2020 08:30:44 +0000) it happened Ahem A Rivet's Shot wrote in :

Oh well I have recompiled kernels on my PCs many many times, not on Raspi, should work the same though. On the PC I could add an entry to grub so I have a selection at boot time what version I run.

But that is also why I do not do upgrades, once you are in that multiple kernel stuff your system is simply unique, and last thing you want is anybody or anything making changes.

My oldest PC still running after ??? years boot menu:

win98 still on it and working last time I looked. Linux 2.6 something kernel choices. I still use it, as it has a satellite TV PCI card and 2 special HQ audio cards and a parport connector (useful for my PIC programmer) Sometimes you need to bang it for it to stop beeping in despair and actually start.. Need to re-insert the board connectors likely, bad contacts).

It is done that way - the Gentoo community have been doing it on the Pi for almost 8 years now and on other platforms for a lot longer.

See

Gordon

I recognized references to kernel compiling, but didn't see any obvious references to non-kernel sources.

Thanks for a most interesting link!

bob prohaska

That's the nail of the thumb 8-) I'm very familiar with FreeBSD and now wondering what alternate options exist for self-hosted systems.

Thanks for writing!

bob prohaska

Perhaps that requires some (inside?) knowledge of what Gentoo is.

Yes, Gentoo can certainly self host and self (re)compile every thing on the system.

I suspect that it can even recompile / relink / reassemble the pieces of proprietary code needed to boot the system. At least given what's publicly available.

--
Grant. . . . 
unix || die

Google buildroot

Near as I can tell that's a cross-compiling toolchain. Couldn't find an obvious reference to self-hosting. What am I missing?

I imagine that with enough work a cross-compiling system could be subverted to self-host, but that seems to be starting the hard way.

Thanks for another interesting link!

bob prohaska

Others have reminded me that Gentoo is a self-hosting Linux distribution - in many ways it looks like FreeBSD will when pkg-base lands. The build system is complex (portage/emerge) but it is self hosting by design like the BSDs.

--
Steve O'Hara-Smith                          |   Directable Mirror Arrays 
C:\>WIN                                     | A better way to focus the sun 
The computer obeys and wins.                |    licences available see 
You lose and Bill collects.                 |    http://www.sohara.org/

In a Debian system, the source for every package is available as a sopurce package. You need the right lines in your /etc/apt/sources.list file. This includes the source for the kernel.

As Raspbian is essentially Debian, I'd expect it to be the same. Indeed on my (very) old out-of-date Raspbian system I can download source files - I simply added a "deb-src" line that the same as my "deb" line, and did

apt-get source some-package-name

in some directory where I want the source files to be stored.

This is documented on the debian wiki at

I read very many responses to your query, but somehow they all seemed to miss this fact! (mumble mumble mumble) from what I've seen I'm not sure asking on the raspberry pi forums would a lot better - but I think you would have got one or two better answers.

Jim

Ok, this is a good start. I find bob@raspberrypi:~ $ more /etc/apt/sources.list deb

buster main contrib non-free rpi # Uncomment line below then 'apt-get update' to enable 'apt-get source' #deb-src buster main contrib non-free rpi

but AIUI apt-get triggers a complete download/overwrite of whatever got downloaded before. That's likely to be wasteful, slow and hard on flash. Is there an option switch like CVS or subversion that fetches changes only?

Thank you very much!

bob prohaska

I saw the references to portage and emerge on the gentoo site and didn't recognize what they did. Alas, Raspbian does not seem to offer them:

bob@raspberrypi:~ $ sudo apt-get install portage [sudo] password for bob: Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package portage bob@raspberrypi:~ $ sudo apt-get install emerge Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package emerge bob@raspberrypi:~ $

Might they reside under another name?

It does appear that raspberrypi.org offers the sources, including encumbered binaries. Paraphrasing a question asked elsewhere, can apt-get (or something else) update in a granular way like subversion or CVS? It seems wasteful to overwrite lots of files to fix a typo.

Thank you very much!

bob prohaska

Try using 'less' instead of 'more'. 'more' does more or less (geddit?) the same job but is nicer to use and can do more useful things, such as starting a new shell over its top or spawning the editor defined by $EDITOR editor so it shows the text that 'less' was displaying. 'exit' the shell or the editor and you're back in 'less', positioned where you left it.

That's what I'd expect, but obligatory warning: I know RH Fedora's 'dnf' a lot better than I know Debian's 'apt'.

--
Martin    | martin at 
Gregorie  | gregorie dot org

I hadn't noticed that more is really more in Raspbian. In FreeBSD more is less 8-) Thanks for pointing the difference out!

Thanks for writing,

bob prohaska

I did make this point as an aside. But it seems the general level of intelligence in IT has been going down over the years. Along with attention span.

--
?People believe certain stories because everyone important tells them,  
and people tell those stories because everyone important believes them.  
Indeed, when a conventional wisdom is at its fullest strength, one?s  
agreement with that conventional wisdom becomes almost a litmus test of  
one?s suitability to be taken seriously.? 

Paul Krugman

bob prohaska wrote:

apt-get source fetches:

- the 'upstream' source code, as a tarball

- the patches applied to the upstream source to make the specific version of the debian package you're fetching

- the control files that allow building the debian package

It'll only download and unpack the things it needs to:

$ apt-get source hello Reading package lists... Done Need to get 734 kB of source archives. Get:1

bionic-updates/main hello 2.10-1build3 (dsc) [1,707 B] Get:2 bionic-updates/main hello 2.10-1build3 (tar) [726 kB] Get:3 bionic-updates/main hello 2.10-1build3 (diff) [6,224 B] Fetched 734 kB in 0s (26.0 MB/s) dpkg-source: info: extracting hello in hello-2.10 dpkg-source: info: unpacking hello_2.10.orig.tar.gz dpkg-source: info: unpacking hello_2.10-1build3.debian.tar.xz

$ apt-get source hello Reading package lists... Done Skipping already downloaded file 'hello_2.10-1build3.dsc' Skipping already downloaded file 'hello_2.10.orig.tar.gz' Skipping already downloaded file 'hello_2.10-1build3.debian.tar.xz' Need to get 0 B of source archives. Skipping unpack of already unpacked source in hello-2.10

You can also ask it to compile the sources:

$ apt-get source --compile hello Reading package lists... Done Skipping already downloaded file 'hello_2.10-1build3.dsc' Skipping already downloaded file 'hello_2.10.orig.tar.gz' Skipping already downloaded file 'hello_2.10-1build3.debian.tar.xz' Need to get 0 B of source archives. Skipping unpack of already unpacked source in hello-2.10 dpkg-buildpackage: info: source package hello dpkg-buildpackage: info: source version 2.10-1build3 dpkg-buildpackage: info: source distribution bionic-security dpkg-buildpackage: info: source changed by Marc Deslauriers dpkg-buildpackage: info: host architecture amd64 dpkg-source --before-build hello-2.10 fakeroot debian/rules clean dh clean debian/rules override_dh_auto_clean make[1]: Entering directory '/tmp/hello-2.10' [ ! -f Makefile ] || /usr/bin/make distclean make[1]: Leaving directory '/tmp/hello-2.10' dh_clean debian/rules build dh build dh_update_autotools_config dh_auto_configure ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=\${prefix}/include --mandir=\${prefix}/share/man --infodir=\${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=\${prefix}/lib/x86_64-linux-gnu --libexecdir=\${prefix}/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking configure: WARNING: unrecognized options: --disable-maintainer-mode checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p [etc]

(turns out 'GNU hello' is a surprisingly complicated package)

Theo

Well yes, I use buildroot running on bigiron Linux machines to cross-compile for customer platforms. But you should be able to set it to run on a Pi for a Pi. I haven't tried it but it should be just a case of setting the cross-compiler to be the installed gcc on the Pi.

It takes a while on a 48core Xeon with 256G of ram so I'm not sure how that will work out even on a Pi4. You'd certainly want something better than an SDcard for storage!