Difference between pages "Java Configuration Design Update" and "How to Dev"

(Difference between pages)
 
 
Line 1: Line 1:
This page describes a potential update to the Java support code in Gentoo and Funtoo Linux, with the intention of simplifying <tt>java-config</tt> code and also making it more correct and better integrated with Portage itself. The proposal intends to address deficiencies in the current design of <tt>java-utils-2.eclass</tt> while maintaining its useful features and avoiding a full rewrite. Rather than replace the current system with more code, the goal is to simplify the existing code while retaining functionality, while allowing a graceful migration of advanced functionality inside Portage itself.
+
= How to 'dev' =
 +
__NOTITLE__
  
{{fancytip|Please post comments on the [[{{TALKPAGENAME}}|Discussion]] page.}}
+
{{fancytip|This page will continue to be updated with development-related information... thanks for your interest :)}}
  
== Design Challenges ==
+
This page serves as the official introduction on how to collaborate with our community and help to improve Funtoo Linux.
  
The current Java eclasses have some very interesting and powerful features, yet are lacking in some areas.
+
== Introduction ==
  
=== Dependency Handling ===
+
Thanks for your interest in helping out with Funtoo Linux! Funtoo Linux has existed for several years, and we've been using git since the beginning of the project, but have played with various approaches on how to develop Funtoo Linux in a collaborative fashion. Some approaches have worked, some haven't, and I (Daniel) and others like to periodically experiment with different approaches, so these methods are likely to change in the future.
  
Currently, <tt>java-utils-2.eclass</tt> uses <tt>depend-java-query</tt> to automatically select the most optimal Java VM for building. This is a sophisticated feature that is intended to be 'smart', yet it has some flaws. These flaws are fixable:
+
== How to Develop -- The Quick Summary ==
  
* While JVM auto-selection is smart, it is not always correct -- it doesn't use Portage's API, but instead uses regexes to parse the currently-executing ebuild's <tt>DEPEND</tt> string.
+
Okay, so you want to get involved. How do you do it? Well, here's how we want you to start:
  
* Related to the regex issue, the code in <tt>depend-java-query</tt> is complex as it tries to duplicate some functionality that could be handled better by Portage's API.
+
* First, you need to actually ''use'' Funtoo Linux in some capacity.
 +
* {{CreateAccount}}
 +
* Find things that need fixing on [https://bugs.funtoo.org the bug tracker], and submit fixes for them.
 +
* If you have a new ebuild, then create an issue on [https://bugs.funtoo.org the bug tracker] for it.
 +
* Testing things and finding bugs is also a form of help.
 +
* Help us document stuff on the wiki.
 +
* Hang out in <tt>#funtoo</tt> on freenode and chat with us.
 +
* Learn more about ebuilds by reading our [[Developer Guide]]. Ask questions.
  
This proposal provides a mechanism for the Portage API to be used to always generate a completely correct result. This may seem like a minimal optimization since the existing code is correct ''most'' of the time. But it is duplicative, in that it does a rough approximation of what the Portage API could do absolutely correctly. So there is no good reason to keep it -- less code means more maintainability.
+
If you start doing a good amount of this, you will get noticed and plugged in to our development efforts in a short amount of time.
  
{{fancyimportant|Moreover, there is even a better reason for doing things the right way -- eclasses should as much as possible be able to behave as "first-class citizens" in the world of Portage. This is an important architectural goal, and there are severe costs for not pursuing it, namely having every eclass under the sun re-implement various parts of Portage, albeit poorly, creating much uncessary code and frustration!}}
+
Now that you have the big picture, let's start by doing a quick overview of Funtoo infrastructure and how it (currently) relates to Funtoo Linux development.
  
=== Atoms ===
+
== bugs.funtoo.org ==
  
Currently, <tt>eselect java-vm</tt> and <tt>java-config</tt> have their own "atoms" that they use to identify Java virtual machines, which seems to be <tt>${PN}-${SLOT}</tt>. Yet Portage already has atoms, and we should try to allow support for them over time. This proposal does require there to be some "link" between the existing JVM atom and the Portage atom. This link currently does not exist.
+
Currently, [https://bugs.funtoo.org bugs.funtoo.org] serves as the center of operations for all development. If you're going to collaborate with us, then you should definitely {{CreateAccount}} and familiarize yourself with our bug tracker. We try to ensure that ''all'' work in Funtoo Linux has a corresponding issue opened on the bug tracker, so there is some reference for someone to look to find out ''why'' some change was made.
  
== Proposal ==
+
== The Wiki ==
  
=== java-config Settings ===
+
You are currently reading the wiki, and it is becoming a more and more important part of development. We currently have the ability to define wiki pages for [[:Category:Ebuilds|Ebuilds]], which are used as official documentation for the ebuild. Documentation is a very important part of Funtoo Linux development, and you're strongly encouraged to read about [[Adding an Ebuild to the Wiki]]. Developers who write good documentation gain much respect :)
  
This proposal involves a minor upgrade to the java-config settings for a JVM -- adding a <tt>PORTAGE_ATOM</tt> setting to the file, which allows the <tt>java-config</tt> atom to be linked to the Portage atom:
+
== git.funtoo.org ==
  
{{file|name=/usr/share/java-config-2/vm/oracle-jdk-bin-1.7|desc=Java-config settings|lang=bash|body=
+
All of our master git repositories exist on [http://git.funtoo.org git.funtoo.org].
# Copyright 1999-2011 Gentoo Foundation
+
# Distributed under the terms of the GNU General Public License v2
+
# $Header: /var/cvsroot/gentoo-x86/dev-java/oracle-jdk-bin/files/oracle-jdk-bin-1.7.env,v 1.2 2011/11/17 22:49:56 caster Exp $
+
  
VERSION="Oracle JDK 1.7.0.60"
+
== GitHub ==
JAVA_HOME="/opt/oracle-jdk-bin-1.7.0.60"
+
JDK_HOME="/opt/oracle-jdk-bin-1.7.0.60"
+
JAVAC=${JAVA_HOME}/bin/javac
+
PATH="${JAVA_HOME}/bin:${JAVA_HOME}/jre/bin"
+
ROOTPATH="${JAVA_HOME}/bin:${JAVA_HOME}/jre/bin"
+
LDPATH="${JAVA_HOME}/jre/lib/amd64/:${JAVA_HOME}/jre/lib/amd64/xawt/:${JAVA_HOME}/jre/lib/amd64/server/"
+
MANPATH="/opt/oracle-jdk-bin-1.7.0.60/man"
+
PROVIDES_TYPE="JDK JRE"
+
PROVIDES_VERSION="1.7"
+
BOOTCLASSPATH="${JAVA_HOME}/jre/lib/resources.jar:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/jre/lib/sunrsasign.jar:${JAVA_HOME}/jre/lib/jsse.jar:${JAVA_HOME}/jre/lib/jce.jar:${JAVA_HOME}/jre/lib/charsets.jar:${JAVA_HOME}/jre/classes"
+
GENERATION="2"
+
ENV_VARS="JAVA_HOME JDK_HOME JAVAC PATH ROOTPATH LDPATH MANPATH"
+
VMHANDLE="oracle-jdk-bin-1.7"
+
BUILD_ONLY="FALSE"
+
PORTAGE_ATOM="dev-java/oracle-jdk-bin-1.7.0.60"
+
}}
+
  
With this very minor and easy-to-implement change, new possibilities are available -- namely, for <tt>java-config</tt> code to tap into the Portage API and leverage its advanced dependency handling functionality.
+
We have mirrors of all our git repositories on [http://github.com/funtoo GitHub]. When we push to a repository on git.funtoo.org, it automatically pushes the change to the corresponding repo on GitHub, too.
  
<div style="float: right; width: 40%; margin-left: 1em; margin-top: 1em; margin-bottom: 1em; background-color: #f8f8f8; padding: 0.5em; border-radius: 8px;">
+
This has historically created some confusion, because we don't really (right now) do active development on GitHub, using pull requests. If you have an improvement to Funtoo Linux, we want you to open a bug on our [https://bugs.funtoo.org bug tracker]. However, I am hoping that this will change - see {{Bug|FL-1322}}. But for now, if you create an issue or a pull request on GitHub, we probably won't see it in a timely manner.
  
=== Provides ===
+
== The Git Repositories Themselves ==
  
{{fancynote|This section is not a requirement for the implementation of this proposal, but a suggestion for future Portage development.}}
+
One important thing to understand about Funtoo Linux is that the git tree you receive when you type <tt>emerge --sync</tt> is what I call a ''transport tree'' -- it is used to deliver updates to you, but it is not used for development. Our main <tt>/usr/portage</tt> tree can be seen at http://git.funtoo.org/ports-2012/tree or http://github.com/funtoo/ports-2012 is generated using automated scripts which take the Gentoo Portage tree, various funtoo overlays, and several other overlays and combine them into a unified tree. We generate an updated commit for our transport tree every 4 hours.
  
Currently, the functionality that is provided by each JVM is stored in the <tt>java-config</tt> file shown above, namely in the <tt>PROVIDES_TYPE</tt> and <tt>PROVIDES_VERSION</tt> variables. This works for us, though it is unfortunate that the (in my opinion) not completely thought-out [http://wiki.gentoo.org/wiki/GLEP:37 GLEP 37] was implemented, which deprecated the <tt>PROVIDES</tt> variable in Portage. This is a useful variable because it gave us a "link" from the installed package to the virtual it provided. No such link currently exists, which is a reduction in useful functionality.
+
=== Development Trees ===
  
It is recommended that <tt>PROVIDES</tt> is un-deprecated in Portage, and used by ebuilds solely for recording what virtuals are provided by the ebuild, so that they can be queried later once the package is installed. This would allow <tt>PROVIDES_TYPE</tt> and <tt>PROVIDES_VERSION</tt> -- functionality duplicated by the Java tools -- to be removed, further reducing the complexity of the Java tools codebase.
+
The repositories that we actually do development in are:
</div>
+
  
=== java-config Dependency Handling ===
+
{{#ask:[[Category:Repositories]] [[Repository Type::Development Tree]]
 +
|? Repository ID}}
  
With these changes, <tt>java-config</tt> and <tt>depend-java-query</tt> to tap directly into the power of Portage dependency handling. Let's see how.
+
[[Repository:Funtoo Overlay|funtoo-overlay]] is where nearly all of the core Funtoo forked ebuilds live. This tree is maintained by senior Funtoo staff and the BDFL, and also contains stuff like our [[Funtoo 1.0 Profile]] system, and the actual scripts that are used to generate our main "transport" tree that users receive.
  
=== Query: List installed VMs ===
+
[[Repository:Funtoo LDAP Overlay|The Funtoo LDAP]] and [[Repository:Funtoo GNOME Overlay|GNOME]] overlays are examples of development trees where Funtoo staff have collected a bunch of ebuilds that were cluttering up funtoo-overlay, and placed them in their own repository to improve organization. These are treated as extensions of funtoo-overlay.
  
To list installed VMs, <tt>java-config</tt> could look in <tt>/usr/share/java-config-2/vm/</tt> to get a list of all installed Java VMs. However, thanks to the <tt>PORTAGE_ATOM</tt> variable, it can then compile a list of corresponding package atoms in <tt>/var/db/pkg</tt>.
+
[[Repository:Funtoo Plex Overlay|The Plex overlay]] and [[Repository:Funtoo DeaDBeeF Overlay|DeaDBeeF]] overlays are examples of overlays that are maintained by independent developers, and these overlays are hosted in these developers' GitHub accounts. We have added these overlays to our merge script, so any updates made to these GitHub repositories are automatically pulled into our transport tree.
  
=== Query: Is a Suitable or Best VM Selected? ===
+
So as you can see, there is quite a bit of flexiblity in how updates can be added to our tree. Senior staff can work on the funtoo-overlay, others can work on the more independent GNOME and LDAP overlays, and (mostly) independent developers can happily maintain ebuilds in their own independent overlays, hosted on GitHub.
 
+
Currently, the <tt>java-utils-2.eclass</tt> attempts to magically select the proper VM for building based on the contents of the <tt>DEPEND</tt> string. This is how this functionality would be implemented correctly, using the Portage API rather than custom code.
+
 
+
To perform this query correctly and efficiently, <tt>depend-java-query</tt> can compile a list of Portage atoms for all installed Java VMs, along with what virtual they provide using <tt>PORTAGE_ATOM</tt>, <tt>PROVIDES_TYPE</tt> and <tt>PROVIDES_VERSION</tt>. Then, the <tt>DEPEND</tt> handed to it can be correctly evaluated using Portage functions. This is how the Portage side of the algorithm would work:
+
 
+
# Pre-process <tt>DEPEND</tt> string:
+
## Remove all components that are not related to <tt>virtual/jdk</tt> and <tt>virtual/jre</tt>.
+
## Evaluate <tt>DEPEND</tt> string using currently-active USE settings to remove all conditionals.
+
# Create a <tt>fakedbapi</tt> and <tt>cpv_inject()</tt> the current system VM Portage atom into it.
+
# Evaluate the pre-processed <tt>DEPEND</tt> string and see if the dependencies are satisfied.
+
# If so, a suitable VM is selected.
+
 
+
To find the "best match" VM, a similar process would be followed. Instead of <tt>cpv_inject</tt>ing the current system VM, ''all'' installed VMs would be injected and a best match would be found.
+
 
+
== Future Directions ==
+
 
+
=== Reusing Code, Merging into Portage ===
+
 
+
Since this functionality could use useful to other eclasses, this could be used as a test for a more generic helper function that could be integrated into Portage. This would allow other eclasses to implement similar functionality without having to have their own custom helper applications.
+
 
+
The <tt>java-config</tt> settings file could be merged into <tt>/var/db/pkg</tt> and a simple API could be added to <tt>vartree</tt> and <tt>dblink</tt> to provide API access to it. This would provide a toolkit for eclasses for storing extra configuration settings with installed ebuilds, which could be useful for a variety of purposes.
+
 
+
All these goals support the idea of code re-use and maintainability, and addressing the problem at the correct architectural level.
+
 
+
[[Category:Portage]]
+
[[Category:FLOP]]
+
__NOEDITSECTION__
+

Revision as of 03:28, 30 June 2014

How to 'dev'

Tip

This page will continue to be updated with development-related information... thanks for your interest :)

This page serves as the official introduction on how to collaborate with our community and help to improve Funtoo Linux.

Introduction

Thanks for your interest in helping out with Funtoo Linux! Funtoo Linux has existed for several years, and we've been using git since the beginning of the project, but have played with various approaches on how to develop Funtoo Linux in a collaborative fashion. Some approaches have worked, some haven't, and I (Daniel) and others like to periodically experiment with different approaches, so these methods are likely to change in the future.

How to Develop -- The Quick Summary

Okay, so you want to get involved. How do you do it? Well, here's how we want you to start:

  • First, you need to actually use Funtoo Linux in some capacity.
  • Create a Funtoo account
  • Find things that need fixing on the bug tracker, and submit fixes for them.
  • If you have a new ebuild, then create an issue on the bug tracker for it.
  • Testing things and finding bugs is also a form of help.
  • Help us document stuff on the wiki.
  • Hang out in #funtoo on freenode and chat with us.
  • Learn more about ebuilds by reading our Developer Guide. Ask questions.

If you start doing a good amount of this, you will get noticed and plugged in to our development efforts in a short amount of time.

Now that you have the big picture, let's start by doing a quick overview of Funtoo infrastructure and how it (currently) relates to Funtoo Linux development.

bugs.funtoo.org

Currently, bugs.funtoo.org serves as the center of operations for all development. If you're going to collaborate with us, then you should definitely Create a Funtoo account and familiarize yourself with our bug tracker. We try to ensure that all work in Funtoo Linux has a corresponding issue opened on the bug tracker, so there is some reference for someone to look to find out why some change was made.

The Wiki

You are currently reading the wiki, and it is becoming a more and more important part of development. We currently have the ability to define wiki pages for Ebuilds, which are used as official documentation for the ebuild. Documentation is a very important part of Funtoo Linux development, and you're strongly encouraged to read about Adding an Ebuild to the Wiki. Developers who write good documentation gain much respect :)

git.funtoo.org

All of our master git repositories exist on git.funtoo.org.

GitHub

We have mirrors of all our git repositories on GitHub. When we push to a repository on git.funtoo.org, it automatically pushes the change to the corresponding repo on GitHub, too.

This has historically created some confusion, because we don't really (right now) do active development on GitHub, using pull requests. If you have an improvement to Funtoo Linux, we want you to open a bug on our bug tracker. However, I am hoping that this will change - see FL-1322. But for now, if you create an issue or a pull request on GitHub, we probably won't see it in a timely manner.

The Git Repositories Themselves

One important thing to understand about Funtoo Linux is that the git tree you receive when you type emerge --sync is what I call a transport tree -- it is used to deliver updates to you, but it is not used for development. Our main /usr/portage tree can be seen at http://git.funtoo.org/ports-2012/tree or http://github.com/funtoo/ports-2012 is generated using automated scripts which take the Gentoo Portage tree, various funtoo overlays, and several other overlays and combine them into a unified tree. We generate an updated commit for our transport tree every 4 hours.

Development Trees

The repositories that we actually do development in are:

 Repository ID
Bar Overlaybar-overlay
Bliss Overlaybliss-overlay
Foo Overlayfoo-overlay
Funtoo DeaDBeeF Overlayfuntoo-deadbeef
Funtoo GNOME Overlayfuntoo-gnome-overlay
Funtoo LDAP Overlayfuntoo-ldap
Funtoo MySQL Overlayfuntoo-mysql
Funtoo Overlayfuntoo-overlay
Funtoo Plex Overlayfuntoo-plex
Progress Overlayprogress
Squeezebox Overlaysqueezebox

funtoo-overlay is where nearly all of the core Funtoo forked ebuilds live. This tree is maintained by senior Funtoo staff and the BDFL, and also contains stuff like our Funtoo 1.0 Profile system, and the actual scripts that are used to generate our main "transport" tree that users receive.

The Funtoo LDAP and GNOME overlays are examples of development trees where Funtoo staff have collected a bunch of ebuilds that were cluttering up funtoo-overlay, and placed them in their own repository to improve organization. These are treated as extensions of funtoo-overlay.

The Plex overlay and DeaDBeeF overlays are examples of overlays that are maintained by independent developers, and these overlays are hosted in these developers' GitHub accounts. We have added these overlays to our merge script, so any updates made to these GitHub repositories are automatically pulled into our transport tree.

So as you can see, there is quite a bit of flexiblity in how updates can be added to our tree. Senior staff can work on the funtoo-overlay, others can work on the more independent GNOME and LDAP overlays, and (mostly) independent developers can happily maintain ebuilds in their own independent overlays, hosted on GitHub.