Background

Some time after installing Gentoo, I moved away from syncing the portage tree via Git, because having history of the repository is nice and as a bonus, the repository can be synchronized from GitHub over HTTPS while rsync uses plain text. Commits in the repository are also GPG-signed, but the current version of portage does not use this (edit: by default, I rechecked the dates and 2.1.24 should have it) to validate the repository (yet)

However, the official Gentoo package repository is not enough, and even though I use Bedrock Linux, it's nice to have packages designed for Gentoo so I use a number of unofficial repositories or overlays to add more packages and the conventional tool to use is Layman which has a decent-enough interface and support for a wide variety of methods to sync an overlay - all of the commonly-known OSS version control systems and some others. These overlays are updated at the same time as the official repository via portage (which calls down to layman to do the actual sync).

When I switched to syncing the tree via Git, I used a repository that had pre-generated all the additional information not directly contained in the actual 'gentoo' repository like metadata and other files. This has the advantage of being simpler to use, but as a consequence the repository's history and disk-usage is bloated because it has to track the entire history of not only the ebuilds and related files, but also the easily-reproducible metadata and files taken from other repos - and recently I found a package that resolves the problem with ease-of-use so I decided to try it out (I had other reasons, but that's for later)

At the same time, I thought that since most of my used overlays were using git, wouldn't it make sense to sync them the same way, rather than going through layman? It'd certainly be simpler, a quick look showed that there was a package that provided a good interface for this, eselect-repository that also was much simpler than layman, being under 500 total lines in a file compared to layman's roughly 10kloc (estimated by cloc after removing autoconf and doc). Any non-git overlays would stay managed by layman, though.

Doing the move

(Note: Some parts left out or tweaked for the purposes of this post.)

1. I added the mv overlay to add the package with the scripts, and then installed both it and eselect-repository with emerge.

2. Ensure /etc/portage/repos.conf/gentoo.conflooks like this (change the clone-depth if you don't want a full clone):

    [DEFAULT]
    main-repo = gentoo
   
    [gentoo]
    location = /usr/portage
    sync-type = git
    sync-uri = https://github.com/gentoo/gentoo.git
    auto-sync = yes
    clone-depth = 0
   

3. Tweak /etc/eselect/repository.conf so it uses the original sources, rather than the ones with metadata in the history.

4. Make a note of your layman-managed git overlays somewhere to re-add them shortly afterwards.
5. Remove (or disable) all the git overlays using layman (-d or -D)
6. Re-add the overlays using eselect repository enable for overlays on the main list, and add for those not
7. Run your first sync command. Ignore the messages about non-existent directories, they'll go away as each overlay is created when synced.

Unrelated Benefits

While this section isn't related to either eselect-repository or layman, it is related to the changes and why I did them. Essentially, after finding out about a certain overlay which had a large number of packages, I noticed that eix-sync was taking a good amount of time to update the database for this large overlay - looking at another, I noticed that I never generated the metadata cache for any of my overlays. While googling about this, I found the complete set of shell scripts to update this for all overlays and repositories, along with other things (and also found a post pointing out the wastefulness of storing all the metadata in git for the Gentoo repository). The end-result was this post, and the generated metadata cache, while taking up-front CPU time, makes the eix updates faster overall and hopefully will also provide a nice speed boost for portage in general.

Next?

While there are not many next steps, the most obvious one is to add support in portage for the sync methods that Layman provides but Portage itself does not. Even though I'm not an expert in Python, this shouldn't be difficult to implement.

(Addendum: Someone in fact already wrote a module for Mercurial syncing, which is one of the two non-git overlay types I use. You can find it on Gentoo's bugzilla and apply it as a user patch, it works perfectly fine in my experience.)

So, I now have a prevalidated (for now) account over at the Super Dimension Fortress (SDF). I'll most likely validate it later, but not right now. In this post, I'm going to go over my initial experiences and my thoughts about them.

Initial Signup Process

Anyone can start using SDF by making a free account, and there's multiple ways of doing so. There's a simple web form on the SDF website to do it. Other methods include connecting to new@sdf.org using either SSH or Telnet (either works)

I used the SSH method - though my initial choice of when to try proved to be wrong, as the one time it didn't time out it disconnected me very early in the process. Anyway, when SSHing to the system you'll get a series of prompts that ask you questions about your new account. There's the obvious ones about username/password, but it also asks you for some details (name, email, and zip code) and isn't clear if this is required or expected (I don't think it is) or how the information entered will be used and displayed. As such, I only entered my name but am happy to enter an email if needed (I do not have a ZIP code, though. Not American, you see?)

If you exit this user account creation process (two methods are saying N to the final 'are you sure' about it or trying to use an already-taken name) you get dumped into a basic command-line menu. The menu's prompt is 'FEP Command' and while this seems inconsequential, it is in fact a reference to a completely different type of computer than the UNIX and Windows-based systems that are currently dominant. In the 1980s and early 1990s, there were companies that made Lisp Machines, which were and still are very different to modern systems running a UNIX-like or Windows. On these systems, there was a Lisp processor, but in many systems the Lisp CPU was unable to boot the system standalone, so there was an additional processor (or for the later ones, something specially integrated with the main processor) to bootstrap the Lisp one with the rest of the system. Symbolics systems, in particular, had a Front End Processor that perfomed this task (as well as other things like handling full system crashes). SDF is not a Lisp Machine, it has no actual FEP. However, this restricted initial commandline serves similar purposes.

• It acts as a minimal 'frontend' to the actual SDF boxen.
• It's used to create new accounts to the system, 'bootstrapping' your access

It's a nice reference or play on words, isn't it?

On the System

As a prevalidated user, you don't get access to a actual shell, instead you get their custom restricted shell, called 'psh'. It has various limitations which will quickly become apparent, especially if you use a very nicely customised zsh environment. A minor detail I noticed is that both 'emacs' and 'nano' end up presenting the same editor: UW Pico 4.10. However, even though the shell is limited you still have the Gopher client and limited usage of lynx. There's also the bulletin board and chat systems, both of which I believe are unique to SDF and sdf-eu. While the FEP prompt had a 'software' command, there's also one in psh and surprisingly the results are very different to the ones you get from the same FEP command (much newer, for example). At the time of writing I haven't heavily used either Gopher, the BBS or the chat system, but hopefully that will change (and this post will be on my new SDF gopherhole too).