Difference between pages "News:Newsletter, Volume 1" and "Repository Configuration"

From Funtoo
(Difference between pages)
Jump to navigation Jump to search
 
(Undo revision 9202 by Duncan.britton (talk))
 
Line 1: Line 1:
{{News
{{Warning|This article is a work-in-progress referring to a future Portage version. It does not apply to the current Funtoo Portage version. Please do not update your configuration yet.}}
|Summary=Discussed: ati-drivers, GitHub integration, Funtoo on ARM, GNOME updates, Organizations, and two new devs.
|News Format=Extended
|Author=Drobbins
|Publication Status=Draft
|Publication Date=2015/01/27
}}
Hello, everyone -- hope your 2015 is off to a good start. Welcome to our first official full newsletter. Funtoo Linux development is ramping up, and with that we have more to discuss. So let's dive in.


=== Ati-drivers updates ===
Starting with Portage-2.3.8, a switch to a new repository configuration framework is complete and users may want to update their configuration files. This document aims to describe the goals for the new framework and how to use it.
[[File:Jetson.jpg|thumb|500px|class=img-responsive|The passively-cooled Funtoo Linux NVIDIA Jetson TK1 ARM development board. The heat sink doesn't even get warm!]]
For those who use [[Package:AMD_Catalyst_Video_Drivers|AMD Catalyst video drivers]] under Funtoo Linux, you likely know that older versions of ati-drivers have not been compatible with the most recent version of xorg-server. Ati-drivers 14.9 required masking xorg-server-1.16, as it would only work with 1.15 or earlier. About a month ago, we received a tip that Arch and Ubuntu had a version that worked with xorg-server-1.16, and upon investigation, found that AMD rolled out an Ubuntu-specific version of their video drivers for the sole purpose of working with xorg-server-1.16. I looked into these drivers and bundled them up at ati-drivers-14.9-r1 in Funtoo Linux.  


However, while they worked with xorg-server-1.16, it turns out that these new drivers were quite flaky, and a number of people had problems with them, including myself. I ended up switching to the free Radeon drivers, which allowed me to avoid the whole ati-drivers mess.
== Multiple repository layout ==
One of the most important changes is the switch from the old ''overlay'' layout to a new cleaner ''repository'' system. The new layout is more flexible and more predictable. For example, repositories can now use resources (eclasses, for example) provided by other repositories.


The good news is that things have gotten better on the AMD Catalyst front. AMD has released a new version of their drivers which is now bundled up in Funtoo Linux as ati-drivers-14.12-r3, and this ebuild does seem quite reliable ''and'' supports xorg-server-1.16. In addition, I have added AMD's GLESv2 and EGL libraries to this ebuild -- up until now, these libraries (distributed separately) have been missing, meaning that many ebuilds including GNOME would use Mesa's GLESv2 implementation, even with ati-drivers enabled. This could potentially lead to issues, so ati-drivers should now be more reliable. In addition, this fixes a bug related to media-libs/cogl and its dynamic loading of GL-related libraries.
The old layout was based on the concept of one ''main tree'' and optionally a number of overlays. The main tree provided base system ebuilds, eclasses, profiles, while overlays mostly were able to provide their own ebuilds. The ebuild provided by overlays overrode the ebuilds in main tree to the extend of making it impossible to install the main tree version. Overlays could also provide eclasses for their own ebuilds and package.* entries that applied to all overlays and to the main tree. The Package Manager is responsible for updating the main tree, while overlays are managed externally.


=== Funtoo Linux for ARM ===
The new layout is based on the concept of one or more configurable repositories. Each repository can either be stand-alone or depend upon other repositories. The distribution provides a repository called ''funtoo'' (a drop-in replacement for Gentoo's ''gentoo'' repository). Users can install more repositories at they will, the repositories providing their own ebuilds, eclasses and profiles as necessary and/or using them from other repositories. Users can explicitly choose the repository they want to install packages from. The Package Manager can update all repositories.


Next up: a quick update on Funtoo Linux for ARM. I've recently moved our ARM builds in-house. Here you can see my passively-cooled NVIDIA Jetson TK1 ARM development board. I had a left-over PC Thermaltake heatsink lying around and decided to put it to use:
== Portage configuration ==
=== New repository layout ===
The repository configuration should be stored in <code>/etc/portage/repos.conf</code>. It can be either a single file or a directory containing one or more ''.conf'' files.


The default configuration is installed as <code>/usr/share/portage/config/repos.conf</code>. This file is internal configuration file installed with portage ebuild and should '''not''' be modified. Instead, the configuration in <code>/etc/portage/repos.conf</code> can override the defaults specified there.


The configuration uses format similar to Windows .ini files. Each section heading (repository name in square brackets) signifies a single repository, followed by one or more key-value option pairs. For example, the following file copies default configuration for Funtoo repository:


While we are now building ARM in-house, I do need to acquire more ARM hardware so that I can actively test ARM builds. That is something that I'd like to resolve soon.
{{file|name=/etc/portage/repos.conf/funtoo.conf|desc=Example configuration override for Funtoo repository to move it to non-standard location|body=
[funtoo]
# moved to non-standard location!
location = /var/db/repos/funtoo
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git
auto-sync = yes
}}


=== Subarches Update ===
The most useful repository configuration options are listed below:
;location: ''Obligatory.'' Specifies the directory where repository is/will be stored. If Portage knows how to sync the repository and the location does not exist, it will be created on next ''emerge --sync''. Otherwise, the directory must exist.
;priority: Specifies the priority used for ordering ebuilds from different repositories. If two repositories provide an ebuild with matching versions, the repository with higher priority will be used.
;auto-sync: Specifies whether ''emerge --sync'' should update the repository. Defaults to ''yes'' if ''sync-type'' is specified, ''no'' otherwise.
;sync-depth: Specifies ''--depth'' for git clone. Used only on initial sync. Defaults to 1. Can be set to 0 to force full clone (not pass ''--depth'' at all).
;sync-type: Specifies syncing/update method. Can be one of: ''cvs'', ''git'', ''rsync'', ''svn''.
;sync-umask: Specifies the umask used when updating/syncing the repository.
;sync-uri: Specifies remote URI from which the repository will be cloned/synced. Can use any syntax valid for a particular syncing method.
;sync-user: Specifies the user[:group] used to update/sync the repository. If ''FEATURES=usersync'' is used, defaults to the credentials of directory owner.


Our new [[Subarches]] page has a well-received improvement for finding the best version of Funtoo Linux for your system, providing a mini-encyclopedia of the different types of hardware that Funtoo Linux supports. I'm always open to adding new stage3 upon request, and Blackadder720 on #funtoo IRC requested an Intel Ivy Bridge build, and this was easy enough to add to our build rotation: [[Intel64-ivybridge]]. So, there you go. You ask for it, and you get it :) Also added -- our subarches page now provides links to our OSUOSL mirror, and I updated the mirroring code so that the OSUOSL links are always fresh.
Additionally a <code>[DEFAULT]</code> section may be specified. Options in this section are used as defaults for all repositories.


=== GitHub Integration (beta) ===
=== Migrating existing configurations ===
The new configuration format provides replacement for existing configuration done through <code>/etc/portage/make.conf</code> and environment variables. While the variables are still supported for backwards compatibility, users are recommended to move to the new configuration scheme. Funtoo portage ebuild is planned to  make the migration unattended (repos.conf installed automatically to ease the config steps) with the following file:
{{file|name=/etc/portage/repos.conf/funtoo.conf|body=
[funtoo]
location = /usr/portage
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git
auto-sync = yes
}}
The following replacements are provided for existing variables:
;PORTDIR: Used to specify main tree location. Replaced by ''location'' key in the section corresponding to the default repository (<code>[funtoo]</code> by default).
;PORTDIR_OVERLAY: Used to specify locations of overlays. Each of the paths needs to be replaced with a separate repository section, with the path placed in ''location'' key. Additionally, ''priority'' may be used to force specific ordering of ebuild overrides.
;SYNC: Used to specify URI for syncing the main repository, also implied a protocol for doing that. Replaced by the ''sync-uri'' and ''sync-type'' keys in the default repository section.
;SYNC_UMASK: Used to specify umask for syncing repositories. Replaced by ''sync-umask'' key in repository configuration. Can be specified in <code>[DEFAULT]</code> section to apply to all repositories.
;SYNC_USER: Used to specify user credentials for syncing repositories. Replaced by ''sync-user'' key in repository configuration. Can be specified in <code>[DEFAULT]</code> section to apply to all repositories.


Thanks to some initiative by Fransisc Simon, we now have some functioning Python code to implement GitHub integration! Here's how it works. If you open an issue or pull request against one of Funtoo Linux's GitHub repositories, our script will import it into our JIRA bug tracker. A friendly note will be posted to your pull request or issue indicating that it has been imported, and a link to the JIRA issue will be provided. When we close an imported issue in JIRA, the script will automatically close the associated JIRA bug.
{{file|name=/etc/portage/make.conf|desc=Example old make.conf file|body=
# user changed PORTDIR location
PORTDIR="/var/db/repos/funtoo"
PORTDIR_OVERLAY="/var/db/repos/foo /var/db/repos/bar"


This is actually a big deal -- how to properly integrate with GitHub has been a long-standing problem. I didn't want to turn Funtoo Linux into a pure GitHub project -- we needed our own advanced bug tracker and independent git repositories. However, many people would like to interact with our development efforts via GitHub directly, and the fact that we mirror our repositories to GitHub tends to invite that.
SYNC="git://github.com/funtoo/ports-2015.git"
SYNC_USER="oleg"
SYNC_UMASK="022"
}}


In the past, pull requests and issues would inevitably be created on GitHub, and these issues would tend to get ignored or forgotten about. Well, thanks to this new script, this is no longer the case. Thanks very much to Francisc for getting this going and I will continue to be working on our GitHub integration and making it more of an integral part of the development process.
{{file|name=/etc/portage/repos.conf|desc=Replacement repos.conf file|body=
[DEFAULT]
sync-user = oleg
sync-umask = 022


=== GNOME Version Bumps ===
[funtoo]
location = /var/db/repos/funtoo
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git


Over the last week, we are starting to slightly bump versions of GNOME packages. These are minor updates, but you should have been noticing them if you are running GNOME and regularly updating your system. If you notice anything unpleasant from these updates, please be sure to [[Reporting Bugs|report a bug]].
[foo]
location = /var/db/repos/foo
priority = 1


=== Media Mix-Ins Update ===
[bar]
location = /var/db/repos/bar
priority = 2
}}


I hope you have had a chance to explore the new functionality in of our [[News:New_Media_Mix-ins|new media mix-ins]]. In general, feedback has been positive. {{user|Digifuzzy}} and {{user|nrc}} on the forums were concerned that in rolling out the new media mix-ins, we did enable some new USE file-format-related USE variables in the desktop profile, which on their systems resulted in a significant increase in the number of packages installed after an {{c|emerge -auDN @world}}.  
The <code>repos.conf</code> configuration can be further extended with ''sync-type'' and ''sync-uri'' for overlays to get ''emerge --sync'' updating them automatically.


There was a suggestion made that we do a better job of informing users of potentially highly-impacting changes ''prior'' to them hitting the tree, and this is something that I will attempt to do a better job of moving forward. What we did not realize is the relatively large impact that a small number of USE variables additions could have for already-installed systems.
let's see a real example of tree and overlays added.
{{file|name=/etc/portage/repos.conf|desc=Replacement repos.conf file|body=


=== Funtoo-Friendly Organizations ===
[gentoo]
 
location = /usr/portage
You may have noticed the [[Organization:Brownrice Internet|Brownrice Internet]] logo on the bottom of our main page. Brownrice Internet is the first organization to take advantage of our Funtoo-Friendly Organizations feature on the wiki. This allows any organization, whether for-profit or community-based, to register themselves on the wiki with their own organizational profile page. If your organization uses Funtoo Linux, offers Funtoo Linux as an option for hosting, or builds something based on Funtoo Linux, then we want you to let our user community know about what you do! To do this, head over to [[:Category:Organizations|the Organizations category on the wiki]] and enter your organization's name, and fill out the form. Get the recognition you deserve! :)
sync-type = git
 
sync-uri = git://github.com/funtoo/ports-2012.git
=== Translations Updates ===
 
[funtoo-overlay]
Many thanks to {{user|Aramisqc}} for completing the [[Install/fr|French translation of our install Instructions]]. Tocadotux is currently working on a Brazilian Portuguese translation, and {{user|Oleg}} is working on a Russian translation. I would be interested in getting a German and Italian translation going -- find me on IRC if you are interested in helping. As the translation effort gets under way, I hope to get the translation team more organized, get translation instructions online and also look into developing some tools to assist in keeping translations in sync.
location = /root/git/funtoo-overlay
 
=== Funtoo Linux Containers ===
[funtoo-gnome]
 
location = /root/git/funtoo-gnome-overlay
Our Funtoo Linux Containers continue to be in demand. I will be limiting the number of new containers on our existing hardware to maximize performance for our existing users, and we currently have 23 spots left. If you are interested in supporting Funtoo Linux and having your own highly-capable SSD-based Funtoo Linux container with up to 24 cores, then see our [[Funtoo Hosting]] page and grab one. You can get an extremely capable container for only $15/mo of support, and our higher-end containers are even more impressive.
}}
 
funtoo-overlay and funtoo-gnome-overlay are an overlays added on top of regular portage tree.
=== New Developers ===
[[Category:Portage]]
 
I'd like to personally welcome two developers. {{user|Anak1n}} has returned to us, and is helping us keep various packages up-to-date and squash  bugs. He recently added the [[Package:Ghost_(Blogging_Platform)|Ghost blogging platform]] ebuild.
 
In addition, {{user|Mgorny}} has joined our team. He also serves as a Gentoo developer, and for us, he is working on various complex technical issues, in addition to helping me and Oleg manage the day-to-day work of keeping on top of bug reports.
 
Welcome to the team! And if you see these guys online, be sure to welcome them and thank them for their assistance.
 
And if there is anyone out there would would like to assist us by doing some serious bug squashing, I am currently looking to grow our team and accelerate our development efforts. Ask me ({{c|drobbins}}) or Oleg ({{c|angry_vincent}}) on IRC about how you can help.
 
Well, that's it for our first newsletter. I hope you liked it!
{{NewsFooter}}

Revision as of 04:52, February 27, 2015

   Warning

This article is a work-in-progress referring to a future Portage version. It does not apply to the current Funtoo Portage version. Please do not update your configuration yet.

Starting with Portage-2.3.8, a switch to a new repository configuration framework is complete and users may want to update their configuration files. This document aims to describe the goals for the new framework and how to use it.

Multiple repository layout

One of the most important changes is the switch from the old overlay layout to a new cleaner repository system. The new layout is more flexible and more predictable. For example, repositories can now use resources (eclasses, for example) provided by other repositories.

The old layout was based on the concept of one main tree and optionally a number of overlays. The main tree provided base system ebuilds, eclasses, profiles, while overlays mostly were able to provide their own ebuilds. The ebuild provided by overlays overrode the ebuilds in main tree to the extend of making it impossible to install the main tree version. Overlays could also provide eclasses for their own ebuilds and package.* entries that applied to all overlays and to the main tree. The Package Manager is responsible for updating the main tree, while overlays are managed externally.

The new layout is based on the concept of one or more configurable repositories. Each repository can either be stand-alone or depend upon other repositories. The distribution provides a repository called funtoo (a drop-in replacement for Gentoo's gentoo repository). Users can install more repositories at they will, the repositories providing their own ebuilds, eclasses and profiles as necessary and/or using them from other repositories. Users can explicitly choose the repository they want to install packages from. The Package Manager can update all repositories.

Portage configuration

New repository layout

The repository configuration should be stored in /etc/portage/repos.conf. It can be either a single file or a directory containing one or more .conf files.

The default configuration is installed as /usr/share/portage/config/repos.conf. This file is internal configuration file installed with portage ebuild and should not be modified. Instead, the configuration in /etc/portage/repos.conf can override the defaults specified there.

The configuration uses format similar to Windows .ini files. Each section heading (repository name in square brackets) signifies a single repository, followed by one or more key-value option pairs. For example, the following file copies default configuration for Funtoo repository:

   /etc/portage/repos.conf/funtoo.conf - Example configuration override for Funtoo repository to move it to non-standard location
[funtoo]
# moved to non-standard location!
location = /var/db/repos/funtoo
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git
auto-sync = yes

The most useful repository configuration options are listed below:

location
Obligatory. Specifies the directory where repository is/will be stored. If Portage knows how to sync the repository and the location does not exist, it will be created on next emerge --sync. Otherwise, the directory must exist.
priority
Specifies the priority used for ordering ebuilds from different repositories. If two repositories provide an ebuild with matching versions, the repository with higher priority will be used.
auto-sync
Specifies whether emerge --sync should update the repository. Defaults to yes if sync-type is specified, no otherwise.
sync-depth
Specifies --depth for git clone. Used only on initial sync. Defaults to 1. Can be set to 0 to force full clone (not pass --depth at all).
sync-type
Specifies syncing/update method. Can be one of: cvs, git, rsync, svn.
sync-umask
Specifies the umask used when updating/syncing the repository.
sync-uri
Specifies remote URI from which the repository will be cloned/synced. Can use any syntax valid for a particular syncing method.
sync-user
Specifies the user[:group] used to update/sync the repository. If FEATURES=usersync is used, defaults to the credentials of directory owner.

Additionally a [DEFAULT] section may be specified. Options in this section are used as defaults for all repositories.

Migrating existing configurations

The new configuration format provides replacement for existing configuration done through /etc/portage/make.conf and environment variables. While the variables are still supported for backwards compatibility, users are recommended to move to the new configuration scheme. Funtoo portage ebuild is planned to make the migration unattended (repos.conf installed automatically to ease the config steps) with the following file:

   /etc/portage/repos.conf/funtoo.conf 
[funtoo]
location = /usr/portage
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git
auto-sync = yes

The following replacements are provided for existing variables:

PORTDIR
Used to specify main tree location. Replaced by location key in the section corresponding to the default repository ([funtoo] by default).
PORTDIR_OVERLAY
Used to specify locations of overlays. Each of the paths needs to be replaced with a separate repository section, with the path placed in location key. Additionally, priority may be used to force specific ordering of ebuild overrides.
SYNC
Used to specify URI for syncing the main repository, also implied a protocol for doing that. Replaced by the sync-uri and sync-type keys in the default repository section.
SYNC_UMASK
Used to specify umask for syncing repositories. Replaced by sync-umask key in repository configuration. Can be specified in [DEFAULT] section to apply to all repositories.
SYNC_USER
Used to specify user credentials for syncing repositories. Replaced by sync-user key in repository configuration. Can be specified in [DEFAULT] section to apply to all repositories.
   /etc/portage/make.conf - Example old make.conf file
# user changed PORTDIR location
PORTDIR="/var/db/repos/funtoo"
PORTDIR_OVERLAY="/var/db/repos/foo /var/db/repos/bar"

SYNC="git://github.com/funtoo/ports-2015.git"
SYNC_USER="oleg"
SYNC_UMASK="022"
   /etc/portage/repos.conf - Replacement repos.conf file
[DEFAULT]
sync-user = oleg
sync-umask = 022

[funtoo]
location = /var/db/repos/funtoo
sync-type = git
sync-uri = git://github.com/funtoo/ports-2015.git

[foo]
location = /var/db/repos/foo
priority = 1

[bar]
location = /var/db/repos/bar
priority = 2

The repos.conf configuration can be further extended with sync-type and sync-uri for overlays to get emerge --sync updating them automatically.

let's see a real example of tree and overlays added.

   /etc/portage/repos.conf - Replacement repos.conf file
[gentoo]
location = /usr/portage
sync-type = git
sync-uri = git://github.com/funtoo/ports-2012.git
 
[funtoo-overlay]
location = /root/git/funtoo-overlay
 
[funtoo-gnome]
location = /root/git/funtoo-gnome-overlay

funtoo-overlay and funtoo-gnome-overlay are an overlays added on top of regular portage tree.

Retrieved from "https://www.funtoo.org/index.php?title=News:Newsletter,_Volume_1&oldid=9204"