Jump to navigation Jump to search
Metatools 1.3.0 is a major release which was released on 29 May 2023.
This is a feature release containing a number of new capabilities and improvements.
- Refactor of how we handle the
--immediateoption internally to be more intuitive in the source code.
- Initial implementation of ZeroMQ-based "moonbeam" communications framework to allow child
doitprocesses to communicate with the master
merge-kitsprocess. This will initially be used to implement logging of all issues encountered during the
merge-kitsrun so we can generate a nice summary of problems (see FL-11179). The initial framework has just been added but the logging/reporting functionality is not yet implemented.
- When running
ensure_fetched(), use an
asyncio.wait(0)to allow scheduling/execution of new asyncio tasks. This method often gets hammered with hundreds of new requests and this can stall out existing async tasks (like when a bunch of crates or go modules are getting queued for download all at once.)
- In doit and merge-kits, a large conversion from more thread-oriented to single-process async (with forked subprocesses for external commands) whenever possible, keeping threads only for CPU parallelization for Portage metadata generation. This allows "moonbeam" to be able to send/receive messages efficiently when other stuff is going on.
- Python 3.7 compatibility restored to the codebase (I made a minor change which made the code 3.9+.)
--howdyargument for merge-kits which causes "HOWDY" to be printed every 0.1 seconds from the moonbeam ZeroMQ engine. This is used to test for any issues related to async tasks not being scheduled to run frequently. If you don't see HOWDY printed continuously then some long-running task is blocking the async event loop and this should probably be fixed.
- Fix a 3-year-old bug where the Python USE-optimization code was not generating deterministic results in package.use files.
- Misc fixes to pyhelper to introduce sorting in some areas to reduce randomization (non-deterministic order) of elements in ebuilds.
- Reimplementation of
deepdive. Add an advanced
deepquerythat can actually rewrite
packages.yamlfiles for us to remove unused ebuilds automatically. This is an active area of work and needs some docs and cleanup.
- When specifying assets: for github-1 to grab, add a special keyword
"<source.tar.gz>"literal string which allows you to grab the auto-generated default tarball. There was not previously a way to grab this as well as other assets that were uploaded to a release.
- Support ETag and Last-Modified HTTP 304 responses. This dramatically improves API limits for GitHub, etc.
- FL-11369: tweak to
rust.pyto background and make the cargo update async-compatible.
- Deprecate max_age= parameter for fetching (this was a vestigial thing that was not being used.)
- As part of the work on HTTP 304 support,
spider.http_fetchnow returns a tuple of headers and content. This is necessary so we can extract "Last-Modified" and "ETag" headers and store them in the fetch cache so we can use them for successive requests for the HTTP 304 support.
- Fix an issue with
doitthat is common to all Python programs -- All python programs will attempt to import things from the current working directory if some directory exists with the same name as a module it needs. This is really, really dumb. This caused
doitto fail in python-modules-kit, inside
curated/dev-python, due to the "click" directory existing after first
doitis run, which then caused successive
doits to fail when
httpxtries to import the
- Cleaned up some error output issues.
- FL-11300: attempt to address Tree OOP hierarchy to ensure initialize is available for
AutoCreatedGitTreeclass. (Thanks: borisp)