aboutsummaryrefslogtreecommitdiffstats
path: root/Tools
Commit message (Collapse)AuthorAgeFilesLines
* Use diff -p both when first creating and when updating a patch.naddy2009-02-161-1/+1
| | | | | Previously a patch created by update-patches would be changed again by the next update-patches run.
* Remove 6-exp2 builds, they haven't been used in a whilepav2009-01-222-6/+3
|
* Factor out the code that performs the PORTREVISION bump for one port.gerald2009-01-081-42/+47
| | | | Approved by: edwin
* Reduce per branch code duplicationerwin2008-12-281-41/+16
| | | | Submitted by: pgollucci
* Make sure we don't add ${WRKDIR}.sobomax2008-10-221-4/+4
|
* Handle the rest of the edge cases; drop 5 as obsolete; cleanup.linimon2008-09-181-23/+35
| | | | | | Now re-enabled on production. Approved by: portmgr (self)
* Bring this into the new portbuild world order. While here, try tolinimon2008-09-181-27/+94
| | | | | | suppress hrefs for non-existant files (incomplete). Approved by: portmgr (self)
* RELENG_5 is desupported.linimon2008-09-121-3/+3
| | | | Approved by: portmgr (self)
* Major overhaul:linimon2008-09-071-349/+172
| | | | | | | | | - bring this closer to the default FreeBSD page style - remove unsupported releases - remove the date stamps, which no longer work - remove obsolete commented-out junk Discussed on: portmgr, some time ago
* Add a new script for performing high-level administration of a build.kris2008-08-091-518/+496
| | | | | | | | | | | | | | | list : lists available builds clone : creates a new build by cloning a previous one portsupdate : update a ports tree to the latest ZFS snapshot srcupdate : update a src tree to the latest ZFS snapshot cleanup : clean up or remove a build on the clients destroy : remove a build on the server There is some trickiness here in that various commands either expect to run as root, or expect to run as a ports-* user. For the latter case we can easily use su to proxy as the ports user when running as root; for the former we use the buildproxy to validate and re-execute the command as root.
* Add an option to clean up old distfiles. We don't have enough space tokris2008-08-091-0/+10
| | | | keep dozens of full distfile sets lying around.
* Update the list of openoffice.org ports: remove 1.x and add 3.xerwin2008-07-301-1/+1
|
* Oops, commit the right version of this (adds extra checks that onlykris2008-07-261-0/+5
| | | | expected commands will be proxied)
* * Simple client that passes on a build request to the buildproxykris2008-07-261-0/+32
|
* * Python daemon run as root that proxies privileged build commands forkris2008-07-261-0/+75
| | | | | | | | the ports-* users. Currently it is not possible to delegate management of ZFS filesystems to non-root users, so root privilege is required to manipulate them. We validate the command passed on a local domain socket and re-execute the build script with the requested parameters.
* Script run from cron to regularly update the master ZFS copies of thekris2008-07-261-0/+63
| | | | | | | | | | | | | | ports and source trees. Since we have >=1 consumer of these trees that run frequently but do not insist on up-to-the-second trees, it makes sense to "pre-update" them regularly and then then re-use in all of the consumers, instead of potentially doing several updates simultaneously or on demand. Consumers can clone the ZFS snapshot into their local filesystem which takes a couple of seconds instead of minutes or tens of minutes for the CVS update. We update to a date stamp instead of "." because this avoids ambiguity of commits that happen while the tree update is in progress (unfortunately it's slower).
* Script run from cron on the package clients to report metrics to ganglia.kris2008-07-261-0/+35
| | | | | | | | Currently we collect: * The current and maximum number of vnodes in use * The number of packages built over the past hour
* Simple script to expire ZFS snapshots older than a certain agekris2008-07-261-0/+47
|
* Python script for backing up ZFS filesystems on pointyhat. For eachkris2008-07-261-0/+217
| | | | | | listed filesystem we take a new snapshot each time it is run and if the last full backup was not too long ago, do a compressed incremental backup from the previous backup.
* * Add comment that this is unusedkris2008-07-261-0/+1
|
* * Add commentskris2008-07-261-1/+6
|
* * Cleanupkris2008-07-261-28/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Support a meta-hostname of 'all' for setting up all clients at once. This is better than the old way of running one copy of the script for each client by hand, since it is easier and involves less duplicated work. * We copy in the per-build ports, src, and bindist .tbz files and .md5 checksums, as well as refreshing the build scripts and bindist-$(hostname).tar customization tarball. * The -force switch forces copying of files and re-extraction of the tarballs on the client. This is necessary in order to propagate local changes to the tarballs after the initial client setup (e.g. if you need to change a file in the ports tree, it must be recompressed, redistributed, and re-extracted on the client). * The -queue switch will poll the client's job queue after completion of the setup. This is racy and should only be used when the machine is not currently accepting jobs. * For cleaning up a build the 'build cleanup' command should now be used instead. It calls back into this command but also allows full clenaup of build-local files on the client. TODO: "all" setups are hard on the server since they may spawn dozens of rsyncs at once. A better solution would be to have a worker pool of setup tasks to limit the maximum load.
* * Cleanupkris2008-07-261-28/+23
| | | | | | | | | | * Catch up to build ID directory changes * Make it easier to kill a build by not running dopackages in the background where it is detached from shell job control. Now, sending a termination signal to this process (e.g. ^C) will also kill off the dopackages script and in turn the processes created by it. Some background processes spawned by dopackages, pdispatch, etc, may still remain and need to be killed by hand.
* * Cleanupkris2008-07-261-496/+581
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Catch up to build ID directory changes * Improve usage() * Fix a variety of small bugs * Remove support for -ftp builds: we have not supported direct uploading for many years due to the desire to manually inspect build output for quality * All data associated to a build is now localized in its own directory named according to a build ID: /var/portbuild/${arch}/${branch}/builds/${buildid}, where ${buildid} is the creation time. These are actually ZFS filesystems. * Tasks such as cloning a new build, updating a ZFS snapshot, and cleaning up a build are exported to the "build" script, which can be used independently. * Creating a new build is done by ZFS cloning and takes a couple of seconds since it is copy-on-write (i.e. no data needs to be copied). * Ports and source trees are also cloned from pre-updated ZFS images (updated regularly from the "updatesnap" cron job). In most cases we do not care if we are building a ports tree that is an hour or so old since it will become outdated almost immediately anyway, so no matter what we do there will be times when a port has been fixed by the time the build error is generated by a client. * In case an up-to-the-second tree is desired, the -portscvs and -srccvs switches update the existing ports tree via CVS. * -noports and -nosrc can be used to prevent any automatic changes to the ports tree. This is useful for dealing with local modifications (e.g. for -exp builds), since the default when creating a new build is to replace the previous trees with fresh, pristine trees. If you forget to use this then any local changes that are not also present in other trees will be lost. * By default we keep two builds for each arch/branch pair. These build IDs also may be referred to via "latest" and "previous" symlinks. When creating a new build, the old "previous" build is destroyed by default, unless it was originally created using the -keep switch. This prevents the build from being destroyed automatically. * By default when a build finishes all of the clients are completely cleaned up (i.e. all build data such as ports trees, tarballs, client chroots, etc are deleted). This is needed to save space on the clients. If you expect to *immediately* perform further builds after this one completes, the -nocleanup switch prevents this step. Otherwise they will just be set up again if further builds are scheduled. * Try to parallelize build pre-processing as much as possible, by running jobs in the background wherever possible. In several places we operate on the same parts of the filesystem from multiple jobs, so we can make good use of caching to improve performance * Clients no longer need to be set up explicitly at the start of the build, they will be set up on-demand when the first job is dispatched to them. This allows fast clients or those that already have been set up to begin building ports as soon as possible, while slow clients are set up in the background. It also improves robustness of client recovery, e.g. if the client was offline at the time of build startup but later brought back online. * Optimize copying back in the previous set of restricted packages by hardlinking instead of copying. TODO: The record of failed ports is arch/branch-global still. This is the only thing preventing us from running concurrent builds of the same arch/branch (e.g. while one is stuck building openoffice, the next build can start to keep the cluster busy). The difficulty is that one build from a later ports tree may signal that a build was successful, then a phase 2 build from an earlier ports tree may indicate that it was broken. The solution is probably to migrate this to a real database instead of a flat file, and query it for the set of broken ports as of a certain ports tree date.
* * Catch up to build ID directory changeskris2008-07-261-42/+155
| | | | | | | | | | | | | | | | | | * Clients no longer mount ports/src trees via NFS (even the FreeBSD.org local clients). This was putting too much load on the server and slowing down builds. * Instead ports and src .tbz files are pushed to the clients and unpacked. MD5 checksums are used to verify correctness * -force forces re-extraction of the tarballs even if they exist and appear to be checked out * Also unpack the compressed bindist TODO: When we are not using md or ZFS builds it would be even faster to keep an unpacked copy of the bindist on the scratch filesystem and hardlink the files into the target directory
* * Catch up to build ID directory changeskris2008-07-261-3/+24
| | | | | * Optimize by copying old packages using cpio -dumpl (i.e. create hardlink instead of copying the file).
* * Catch up to build ID directory changeskris2008-07-262-9/+16
|
* * Cleanupkris2008-07-261-50/+51
| | | | | * Catch up to build ID directory changes * Remove need for /etc/arch file
* * Cleanupkris2008-07-261-71/+73
| | | | | | * Catch up to build ID directory changes * Improved support for ZFS * Desupport X11BASE
* * Cleanupkris2008-07-261-75/+120
| | | | | | | | | | | | | * Catch up to build ID directory changes * Improved support for ZFS builds * Improved robustness * Report status verbosely to the caller; whether we succeeded in claiming a chroot, whether the caller needs to first set up the client, or whether a setup is in progress. * If we discover that the client has not been set up either because it freshly booted and newfs'ed its filesystem, or because a particular build has not yet been encountered, atomically claim a cookie and report this to the caller to act on
* * Cleanupkris2008-07-263-24/+50
| | | | * Catch up to build ID directory changes
* * Cleanupkris2008-07-261-19/+74
| | | | | | | * Catch up to build ID directory changes * Add helper functions for resolving a build ID symlink and validating an arch/branch combination (centralize instead of doing it in many scripts)
* * Desupport alpha and ia64kris2008-07-261-4/+3
| | | | * Catch up to build ID directory changes
* * Add commentskris2008-07-261-2/+2
|
* * Implement basename and dirname using shell builtinskris2008-07-261-0/+8
|
* * Cleanupkris2008-07-261-13/+15
| | | | | * Catch up to build ID directory changes * Record package build completion for reporting to ganglia
* * Cleanupkris2008-07-261-104/+160
| | | | | | | | | | | | | | * Catch up to build ID directory changes * Add support for ssh_cmd and scp_cmd to allow using HPN-SSH with the none cipher where possible (for performance) * Lazy client setup; claim-chroot will report if the client needs to be set up with this buildid, and we initiate the setup and poll until it is complete. This allows fast clients to begin building before slow ones have finished setting up. TODO: a better solution would be to avoid trying to dispatch jobs onto clients that are in the process of setting up, since they often have low loads and are picked preferentially by the job scheduler.
* * Cleanupkris2008-07-261-65/+41
| | | | | | | | | | * Remove vestiges of archaic support for building bindists from FTP snapshots; we haven't used this for years and building a world is no longer a challenge * Revert half-baked bindist generation number and make it per-buildid instead. Compress and md5 it for distribution to the clients. TODO: Merge with makeworld?
* * Cleanupkris2008-07-261-17/+24
| | | | | | * Catch up to build ID directory changes * Optimize by using ECHO_MSG=true instead of /usr/bin/true * Try harder to avoid pollution from local host
* * Cleanupkris2008-07-261-7/+18
| | | | * Catch up to build ID directory layout
* * Cleanupkris2008-07-261-51/+18
| | | | | | * Catch up to build ID directory changes * Export the INDEX_PRISTINE and INDEX_QUIET variables (old bug) * Desupport X11BASE
* * Cleanupkris2008-07-261-38/+39
| | | | | * Catch up to build ID directory changes * Desupport 5.x
* * Cleanupkris2008-07-261-15/+17
| | | | | * Catch up to build ID directory changes * Optimize by using __MAKE_SHELL=/rescue/sh
* Rewrite in python and combine the functions of the formerkris2008-07-261-35/+263
| | | | | | | | | checkmachines script. Polls build machines for their status either once-off or regularly as a daemon. Optionally it will update the queue entries but this remains subject to race conditions. TODO: Integrate with queue manager and forward machine status changes to it
* More verbose status reporting using key=value format. We now alsokris2008-07-261-8/+18
| | | | | report error status, architecture and OS version, and available build environments, as well as load and number of running jobs
* Major optimizations for 'make index' and other recursive traversalkris2008-07-201-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | targets. * Use /rescue/sh for index builds instead of /bin/sh, when it exists. The former is statically linked and faster to execute, which becomes significant when executing it tens of thousands of times. This trick can be used with other recursive targets by passing in __MAKE_SHELL. * Get rid of make variable assignments that use != command invocations in the critical path, using several methods: - rewriting logic to use shell or make builtins instead of external command executions - macroizing commands and executing them in the targets where they are needed instead of with every invocation of make - precomputing the results of invariant commands in bsd.port.subdir.mk and passing them in explicitly to child makes, and using this to avoid recalculation in all the children. NB: the commands are still run one per top-level subdirectory but this does not currently seem to be a major issue. They could be moved further up into the top-level Makefile at the cost of some cleanliness. - Committers are strongly discouraged from adding further "bare" != assignments to the ports tree, even in their own ports. One of the above strategies should be used to avoid future bloat. * Rewrite the core 'describe' target to work entirely within a single shell process using only builtin commands. The old version is retained as a backup for use on systems older than 603104, which does not have the make :u modifier. This cuts down the number of processes executed during the course of a 'make index' by an order of magnitude, and we are essentially now amortized to the minimum of a single make + sh instance per port, plus whatever commands the port makefile itself executes (which are usually unnecessary and bogus). * Less validation of the WWW: target is performed; this can become policed at a port level by portlint. Specifically we look at the second word of the first line beginning with "WWW:" in pkg-descr, and append "http://" to it unless it already begins with "http://", "https://" or "ftp://". Thanks to dougb for the idea of how to extract WWW: using shell builtins. * Use the "true" shell builtin instead of echo > /dev/null for a measurable decrease in CPU use. * Add a note about dubious escaping strategy in bsd.port.subdir.mk * Minor change in output of 'make describe': it no longer strips trailing CR characters from pkg-descr files with MSDOS CR/LF termination. Instead the makeindex perl script that post-processes make describe into the INDEX is tweaked to strip on input. The bottom line is that on my test hardware INDEX builds are now faster by more than a factor of 2 and with a reduction in system time by a factor of 4-8 depending on configuration.
* Have a different approach on how to check if the modules file has changed.edwin2008-07-141-9/+7
| | | | | | | | | | Because the $FreeBSD$ keyword isn't expanded in the new version, we can't just do a diff, check the return value and ignore the output. Every new modules file, changed or not with regarding to the contents, has at least four lines in the diff output (line number, old line, seperator, new line). Only commit it if there are more than four lines difference between it.
* Keep the sources of the CVSROOT-ports/modules update script safelyedwin2008-07-094-0/+361
| | | | in the ports CVS repository.
* [PATCH] Tools/scripts: Some cleanups, additions to bump_version.pledwin2008-07-061-28/+33
| | | | | | | | | | | | | | - added -c, doesn't change anything - added -n, deal with an already checkouted tree - removed negative logic in favor of 'unless' - switch to 3 arg form of open() - don't use globs for filehandles, this is been obsolete since at least 5.6.1 - handle possible errors in close() - allow CVSROOT to be overriden in the ENV PR: ports/125025 Submitted by: "Philip M. Gollucci" <pgollucci@p6m7g8.com>
* Reflect latest changes from production:linimon2008-07-021-39/+43
| | | | | | - no more 5-exp - add 8, 8-exp - fix two error-name hrefs
* This conversion script is no longer usefulkris2008-06-261-74/+0
|
* CVSROOT/modules no longer contains an entry per port, remove supporterwin2008-06-204-300/+9
| | | | for this from these scripts.
* Modernize this script a bit.kris2008-06-111-13/+10
| | | | | | | | | | * Remove 5.x support * Leave the archaic ftp snapshot support for now, it is not hurting anything but will not work * Be more careful when removing files (use absolute paths) * Switch to bindist/tmp for the tmp dir * Fix the recording of the bindist.tar generation number * Get rid of redundant or useless processing of the world image
* * Distfile collection is now the default; replace -distfiles with -nodistfileskris2008-06-111-9/+13
| | | | | | * Record the CVS update stamp in some extra places and make sure to remove it if the build is started with -noportscvs (since this probably means the ports tree was updated by hand at some random time)
* Add some test -d's to avoid cd'ing into directories that do not existkris2008-06-111-3/+6
|
* Major optimizations. Instead of copying the distfiles around, mvkris2008-06-111-11/+67
| | | | them in batches according to their target directory.
* Revive this script and make it useful. Transfer the distfiles using rsynckris2008-06-111-26/+23
| | | | and make sure they have been post-processed first.
* * Catch up to X11R6 removalkris2008-06-111-7/+12
| | | | | * Keep RESTRICTED distfiles in a separate DISTDIR so we can easily avoid accidentally publishing them to the FTP site (idea from des@)
* Add support for "PORTREVISION?=" and "PORTEPOCH"edwin2008-06-061-52/+62
| | | | Tested on: devel/gettext bump
* Rewrite this to make it more efficient (fewer external commandkris2008-06-031-138/+181
| | | | | | | invocations). It also fixes some edge cases that were not handled in the previous version. TODO: Correctly report IPv6 sockets (already in use by the sparc64 build)
* Remove support for FreeBSD 5.xerwin2008-06-011-21/+1
|
* Remove the force file, if it was used.linimon2008-05-291-0/+1
| | | | Forgotten by: linimon
* Add a force function here, just like processlogs.linimon2008-05-291-1/+1
|
* Add a note that processonelog and processlogs2 are finicky about thelinimon2008-05-291-0/+1
| | | | header format of the log files.
* Fix these after the 1.31 update to buildscript.linimon2008-05-292-5/+7
| | | | Forgotten by: pav
* Rewrite this in python instead of shell. Because we can read thekris2008-05-261-39/+55
| | | | | INDEX once and process internally instead of invoking many external utilities, runtime is improved from ~20 minutes to <10 seconds.
* NNew build scheduler written in python to replace the make+sh jobkris2008-05-102-0/+1040
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | ordering, which had become too limited. We now build packages ordered by those that are part of the longest dependency chains first. This has the effect of building the deepest parts of the tree first and levelling out the tree height, hopefully avoiding the situation we currently face where there appear bottlenecks late in the build where the cluster becomes mostly idle while waiting for a few long dependency chains to finish building before the cluster can become fully loaded again. The algorithm is that we sort the list of remaining packages according to height (longest dependency chain), then add leaf packages from each in order until we have filled a queue of length between 100 and 200, to amortise the cost of this queue rebalancing while not losing the height averaging property. Jobs are dispatched from this queue into worker threads as machine slots become available. Unlike the make-based solution that required a fixed -j concurrency value and could not respond to addition/removal of build resources, we now can dynamically add new machines as they become available to the queue. The other advantage of using python is that we have more customisability and visibility into the build status, e.g. we periodically report the number of remaining packages, as well as the list of deepest packages that we are working on. TODO: * Implement mtime checking for parent package staleness, so that parents are rebuilt if the dependencies are touched more recently. Currently packages will not be rebuild if they exist, whether or not they are "stale" wrt their dependencies. * Offload the machine selection into an external queue manager. Currently the queue manager used here doesn't interoperate with the old one (getmachine/releasemachine) because it's not possible to use the lockf()-based mutual exclusion within a multithreaded client. Doing that will also allow for a more flexible job placement algorithm as well as finer queue customization.
* Add a tiny sh hack - ardiff - compares two archives.sat2008-05-072-0/+98
|
* Remove XFree86-4 from quickportspav2008-04-111-1/+1
|
* Parallelize to 4 concurrent jobspav2008-04-112-2/+2
|
* Include per-machine configuration and respect use_zfs flagpav2008-04-111-2/+3
|
* Sync with pointyhat (reorg)pav2008-04-111-9/+9
|
* - Add the port's Makefile ident string to the build log headerspav2008-03-131-2/+7
| | | | | | | PR: 113234 Submitted by: Andrej Zverev <andrey.zverev@electro-com.ru> - Sync with actual script on pointyhat (whitespace, mostly)
* - Don't write queue entry for machine that's not on mlist. This allows us topav2008-03-131-0/+6
| | | | | gracefully remove a node from the workload, and also it prevents build from stucking when machine is deleted from mlist.
* Make the INDEX_JOBS definition actually effectivepav2008-03-121-1/+1
|
* Remove support for 5-exp and add 7-exp, 8 and 8-experwin2008-01-155-11/+17
|
* Change homebrew locking routines with lockf(1) based locking.erwin2008-01-011-14/+4
|
* Add the standard 6-line header and URL to all build failure logs, notlinimon2007-12-241-1/+10
| | | | | | | | | just the plist ones. If the log is less than 1000 lines after the header, include it all; else, trim to last 1000 lines. This should help when deciding where to forward logs. Tested on: pointyhat
* Copy the duds file to a directory where the web server can find it. Thislinimon2007-12-241-0/+7
| | | | | | | | makes it possible to correctly analyze why packages were not built for a specific run. Add a beginning and ending email notification to help coordinate between multiple portmgrs doing runs.
* Make the script less whiny for unpopulated builds.linimon2007-12-241-2/+2
|
* Fix a problem when add new ports on ports-mgmt category. ports-mgmt/Makefilegarga2007-12-201-6/+13
| | | | | | | | | | lines has 3 spaces before SUBDIR word and all other categories has 4. I've asked pav@ if there is a default format of category Makefiles and he said the number of spaces doesn't matter, so, i fix addport to respect the current number of spaces and/or tabs the file has. Reported by: miwi, erwin
* Add support for 7-exp and 8 brancheskris2007-11-042-3/+3
| | | | Approved by: portmgr (self)
* Correctly report active builds when ZFS is in usekris2007-11-041-1/+1
| | | | Approved by: portmgr (self)
* * Add support for 7-exp and 8 brancheskris2007-11-041-3/+8
| | | | | | * XORG_UPGRADE and USA_RESIDENT are no longer required Approved by: portmgr (self)
* - Add support for FreeBSD 8.0erwin2007-10-261-4/+25
| | | | | - Be more robust about param.h and allow space as well as tab as seperator while determining OSVERION
* Add 7-exp, 8.linimon2007-10-161-0/+8
|
* Add 7-exp and 8.linimon2007-10-161-1/+1
|
* Rename 'missing' column to 'not yet built' so as not to confuse terminologylinimon2007-10-161-6/+6
| | | | | | with portsmon reports. Noticed by: erwin
* s/query-pr.real/query-pr/ to fix getpr after freefall migrationgarga2007-10-161-1/+1
| | | | Discussed at: #bsdports
* Reflect 8-CURRENT.linimon2007-10-151-1/+94
|
* The index page used on pointyhat.linimon2007-10-151-0/+950
|
* Update this example file to reflect 7.0 branching.linimon2007-10-151-1/+1
|
* Replace two further instances of /var/db/pkg by $PKG_DBDIR.gerald2007-09-171-2/+2
|
* Remove the following false positives, which show up as compiler warnings,linimon2007-09-131-2/+2
| | | | | | | | not errors: discards qualifiers [0-9]: implicit declaration of function ISO C\+\+ forbids
* Use PKG_DBDIR instead of hard-coding /var/db/pkg/. Do not silently revertgerald2007-09-091-3/+15
| | | | | | | to defaults for PORTSDIR and PKG_DBDIR if the respective directory does not exist, but bail out. Approved by: netchild
* Change to the latest names of the ooo ports; add jdk16; sort a bit.linimon2007-09-051-1/+1
| | | | Glanced at by: kris
* explicit_lib_depends.sh:netchild2007-09-053-14/+120
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - check if an installed libtool records dependencies recursively and print a warning if it does currently it prints the warning on every system which has libtool installed from ports (only my local version doesn't do this, the version in the ports is not correctly patched for this, a patch similar in complexity (= simple) like the ltdl.m4 one in the libtool-port-patch-directory is needed) - enhance the regex which is responsible to not print a dependency to the port we are just checking - add a work in progress (not executed) to collapse the USE_* which can have more than one value neededlibs.sh: - we also care about shared libs resolveportsfromlibs.sh: - take care about USE_OPENSSL, USE_EFL, USE_GL, USE_FAM, USE_OPENLDAP, USE_SDL - search in the "ldconfig -r" output if we can not find the lib ourself - a better way of getting the first part of the LIB_DEPENDS stuff (lib/libXYZ.so can be specified now too) - some line wrapping + whitespace - print the origin for the USE_* too (except USE_OPENSSL), so an user can make some sanity checks and the explicit_lib_depends.sh can DTRT if we check the USE_* port itself - warn if we can not determine the right component (can happen for XORG) unambiguously.
* One more sanity check, else /+CONTENS will be searched.netchild2007-08-311-0/+5
|
* USE_FREETYPE is for v1, not for v2. Use a perfect match in the case statement.netchild2007-08-311-4/+5
|
* Do not list libs which come with the port itself.netchild2007-08-261-2/+5
|
* Add explicit_lib_depends.sh.netchild2007-08-261-0/+2
|
* - Add USE_FREETYPE.netchild2007-08-261-1/+5
| | | | | - Print the portsdir variable in front of the origin for easy copy und paste of the output.
* We should also look into sbin/ (not only bin/ and lib/).netchild2007-08-241-1/+1
|
* explicit_lib_depends.sh shows you the current explicit dependency list ofnetchild2007-08-241-0/+115
| | | | | | | | | | | | | | | | | libs for a given installed port, e.g.: # ./explicit_lib_depends.sh gnome-vfs-2.18.1_2 USE_GETTEXT=yes USE_GNOME+=gconf2 USE_GNOME+=glib20 USE_GNOME+=gnomevfs2 USE_GNOME+=libxml2 USE_GNOME+=orbit2 USE_ICONV=yes avahi-client:net/avahi avahi-common:net/avahi avahi-glib:net/avahi dbus-1:devel/dbus dbus-glib-1:devel/dbus-glib
* Teach about USE_ICONV, USE_GETTEXT, USE_GNOME and USE_XORG.netchild2007-08-241-1/+38
|
* Add 2 scripts which help in port maintenance to list all direct librarynetchild2007-08-123-0/+170
| | | | | | | | | | | | | | | dependencies of a port: neededlibs.sh Extract direct library dependencies (filenames) from binaries. resolveportsfromlibs.sh Prints the name(s) of ports(s) given a library filename, suitable for direct use (copy&paste) in LIB_DEPENDS. Example usage is included in the scripts. The following combined usage may be helpful for further porting/testing automation: resolveportsfromlibs.sh -b /usr/local $(neededlibs.sh /test/bin/*) Requested by: kris, lofi (sort of)
* - Extract command variable definitions from bsd.port.mk, they are now stored ingabor2007-08-041-40/+10
| | | | | | | | | bsd.commands.mk and can be easily reused within the infrastructure. - Revert old DESTDIR implementation. - Add a new, fully chrooted DESTDIR implementation as bsd.destdir.mk. Sponsored by: Google Summer of Code 2007 Approved by: portmgr (pav)
* Support builds using zfs and tmpfs backing storekris2007-07-303-7/+40
| | | | | | | | | | | | | | | | | | | | | | | | | zfs: * Enabled by use_zfs=1 in portbuild.conf * Populate build chroots by cloning a zfs snapshot instead of maintaining many duplicate copies. In principle this is very efficient since everything is copy-on-write and zfs snapshot creation is almost instantaneous. There might be additional overheads from building on zfs though. Currently the snapshot base is hard-wired to y/${branch}@base but should be parametrized. This also must be populated beforehand, e.g. during machine startup * Clean build chroots by just destroying the snapshot. tmpfs: * Enabled by use_tmpfs=1 and tmpfs_size in portbuild.conf * The previous md strategy of mounting in used/, populating and then remounting (to avoid possible races from multiple builds claiming the same chroot) doesn't work here because tmpfs instances are destroyed at umount. I am not entirely sure the simpler approach is free from races.
* Refresh old sample portbuild.conf with the one currently used for i386kris2007-07-301-5/+16
| | | | builds. Probably this should be better documented and commented ;)
* * Clean up hacks that had to copy in shared libraries from the host inkris2007-07-301-32/+11
| | | | | | | | | | | | order to run certain host binaries that were kernel-dependent. We now seem to be able to rely on the /rescue versions (and killall(1) seems to be unused). * Allow for ccache directories to be shared over NFS via the ccache_dir_nfs portbuild.conf boolean * Populate BSD.local.dist from ${PORTSDIR}/Templates and remove population of BSD.x11-4.dist and support for XFree86 3.x
* Correct typo ("singal")kris2007-07-301-1/+1
|
* * Switch from netcat port to base system nc. Replace symbolic port namekris2007-07-301-2/+2
| | | | by its numeric equivalent since nc(1) does not look up the former.
* * Retire 4 build and add 6-exp2 experimental buildkris2007-07-301-2/+1
| | | | | * Don't populate BSD.local.dist, this file is maintained in the ports tree now and is populated at build time
* Retire support for 4 and 4-exp buildskris2007-07-301-8/+0
|
* * Increase index build concurrency to reflect the new, improved pointyhatkris2007-07-301-1/+5
| | | | * Add support for 6-exp2 experimental builds
* * Revise the machine selection algorithm. Previously it chose the firstkris2007-07-301-8/+17
| | | | | | | | | | machine with the lowest number of running jobs. This worked when the clients were all roughly equivalent, but schedules poorly when there are some that are much more powerful (e.g. 8-core machines vs UP machines) * We now compute the ratio of running jobs to maximum jobs and schedule on the machine with lowest occupation fraction. This populates the machines to equal fractions of their capacity.
* Use rsync -L instead of rsync -l to deal with the fact that we like tokris2007-07-301-2/+2
| | | | | symlink ports and src trees elsewhere. With -l this would turn a populated tree into a dangling symlink.
* * Remove support for 4 and 4-exp builds and add 6-exp2 experimental buildkris2007-07-301-3/+4
| | | | | | | * Only hardlink the old log files instead of anything else that might be in the directories * Add comment that old logfiles should be removed as well as packages, to avoid duplicate versions of the same port log
* Introduce straslivy.py, a fancier showrunning type of thing, written in Python.pav2007-07-301-0/+147
|
* * Remove processing of 4 and 4-exp logs now that the builds are no longerkris2007-07-301-3/+3
| | | | | | run * Add a 6-exp2 build which is a second experimental tree * Remove an archaic 4.x/5.x log comparison
* * Remove vestiges of procfs mounting/umounting from here; we have tokris2007-07-301-23/+15
| | | | | | | | | | | | | | do it in portbuild from outside the jail thesedays * Ignore /var/db/fontconfig which does not get restored to pristine state * Save copies of master.passwd and groups and check them after the build for changes, to look for user/group additions that may not be correctly registered in UIDs/GIDs. Future work will hopefully automatically check against those files and make unregistered IDs a fatal condition * Correct logic mistake that was keeping distfiles for collection when the checksum mismatched
* - Log the failure to copy dependant package from pointyhatpav2007-07-301-1/+1
|
* Refactor common variables from case statement for various branches.kris2007-07-301-35/+12
| | | | | Update default X11BASE. Retire PARALLEL_PACKAGE_BUILD and HAVE_MOTIF which are now obsolete.
* Retire this obsolete instruction file dating back to last century. Wekris2007-07-301-94/+0
| | | | have a more modern and complete version in the doc collection.
* Drop 4.X statistics. They are half-a-year stale.linimon2007-07-221-1/+2
|
* Add 9 more cases of gcc4; remove some false positives for autoconf; addlinimon2007-07-021-2/+6
| | | | | an 'f77' reason; flag 'warnings being treated as errors' as 'cc'. This reduces the unknown output on i386-7 by about half.
* ARCH may be set in the environment for cross building, so onlyerwin2007-06-121-2/+4
| | | | | | set it conditionally. Submitted by: kris
* Set default architecture to i386.erwin2007-06-121-0/+3
|
* Make it possible to override the templates by using an environementerwin2007-05-251-1/+2
| | | | | | | variable TEMPLATES. RT: 105108 (reworked from) Submitted by: Curtis Jewell<swordsman@csjewell.fastmail.us>
* - Welcome X.org 7.2 \o/.flz2007-05-201-0/+156
| | | | | - Set X11BASE to ${LOCALBASE} for recent ${OSVERSION}. - Bump PORTREVISION for ports intalling files in ${X11BASE}.
* Fix addport to work correctly with ports-mgmt category.garga2007-04-201-1/+1
| | | | | Reported by: miwi Tested by: miwi
* Use echo instead of printf because external commands cannot be executedvd2007-03-291-3/+3
| | | | | | | | | | | with very long arguments (>400000 characters). The problem reveals for example if /usr/ports/Tools/scripts/rmport -d print/ghostscript-gnu is executed - it does printf "%s\n" "... 451109 chars ..." Spotted by: rafan
* Add a column for 'build logs'. This is larger than the number of thelinimon2007-03-231-9/+19
| | | | | | | | packages due to packages being trimmed by RESTRICTED. While here, note that the 'missing' column will be off by the number of duplicates in the other columns. This happens when partial builds are restarted.
* Workaround for the fact that 4/ports is no longer populated on pointyhat.linimon2007-03-091-1/+6
|
* * Run cvs update just before cvs commit to sync a potentially changed filesvd2007-02-191-2/+5
| | | | | in the repository. Suggested by: gabor@ * Fix typo in the output
* Increase resource limits to 400MB filesize and 2 hours CPU limit perkris2007-02-181-5/+5
| | | | | | process. Thankyou openoffice! Pass in the jail IP address in the JAIL_ADDR environment variable
* Default to -nocopy when using -norsynckris2007-02-181-2/+2
|
* Also filter out gconf leftover files from /usr/local as well as /usr/X11R6.kris2007-02-181-4/+5
| | | | | Set DISPLAY to the IP address instead of using the :<...> implied form. This is required for jails.
* Don't require arch to be passed in on the command line, query it usingkris2007-02-181-1/+1
| | | | uname
* Don't play silly games trying to guess the arch, we are passed it in onkris2007-02-181-2/+2
| | | | the command line
* Increase global build timeout to 100 hourskris2007-02-181-8/+2
|
* Improved sanitization of the build environmentkris2007-02-181-0/+6
|
* Don't require an /etc/arch file, we can obtain the information from unamekris2007-02-181-1/+1
|
* Also attempt to clean /root/.ccache in case it is in use. When cleaningkris2007-02-181-7/+5
| | | | | | | on a machine that has use_md_swap=1, allow for the possibility of reusing a md between builds if md_persistent=1. This requires a patch from pjd to support BIO_DELETE in md devices, but it is a big optimization when it can be used.
* Index building needs to happen before prunefailures is run, otherwise itkris2007-02-181-6/+6
| | | | | | | will completely erase the list. When using -continue, don't append to the duds.errors, we don't want to preserve the old cruft from a previous build
* Increase cleanup timeout to 5 days from 48 hours, the openoffice buildskris2007-02-181-3/+3
| | | | now regularly take longer than this :(
* Fix late typo in previous commit.linimon2007-02-141-1/+1
|
* Eliminate some false positives for missing_header, sometimes seen in thelinimon2007-02-141-1/+1
| | | | configure step.
* Combine all like terms to get another 10% speedup in processing time.linimon2007-02-141-116/+13
| | | | | | | | | | There is no change in any of the individual terms; this is merely a rearrangement. This change undoes what I was trying to do back in 2004 of breaking up each individual test into a grep, for readability. The performance of the script has continued to suffer as new greps were added over time, to the point where this is now a bad tradeoff.
* Move two more cases into the 'uncommon' section.linimon2007-02-101-4/+4
|
* Append one character to the gcc4 case (missed during cut-and-paste).linimon2007-02-101-1/+1
|
* Move the uncommon cases below the common ones. A 10% improvement on largelinimon2007-02-101-45/+44
| | | | | | | directories, but a 5% loss on smaller ones. No code changes (yet) except for the deletion of one duplicate ("fetch: transfer timed out" -> "fetch_timeout".
* Remove the other 2 cases of apxs.linimon2007-02-101-4/+0
|
* Speed up by removing the following obsolete tags:linimon2007-02-101-19/+1
| | | | alignment apxs assert CATEGORIES cgi-bin forbidden stl
* Add a new case, 'gcc4', to try to catch the majority of the exceptions thatlinimon2007-02-091-0/+2
| | | | | we are seeing on the latest -current runs. Saves 40 seconds of time on those runs, at the cost of 4 on the others.
* Fix up the code for amd64-6-exp; add a column to label builds thatlinimon2007-02-071-8/+23
| | | | still have processes running, and add that into the highlight logic.
* Retire PORTOBJFORMATkris2007-02-012-2/+1
|
* Drop the build on RELENG_4erwin2007-01-301-20/+1
|
* A bunch of cosmetic changes - add one empty line to make the output morevd2007-01-261-6/+7
| | | | readable and use double quotes everywhere.
* Make output messages more usefulvd2007-01-261-1/+5
|
* * Make it possible to skip existing dependencies and ports references in LEGALvd2007-01-191-39/+67
| | | | | | and continue with removal anyway. Requested by miwi@ * Pipe dependencies information (if any) through a PAGER because INDEX lines are very long and hard to read when wrapped
* Eliminate more false positives for install_error. Very noticeable onlinimon2006-12-261-1/+1
| | | | latest gcc test build.
* Query Gnats for PRs by using the query-pr command on freefall instead of thevd2006-12-201-2/+22
| | | | web-interface becase the later is not reliable.
* Add check for existing non-closed PRs related to the port being removed.vd2006-12-131-1/+65
| | | | | If any are found then ask if the port should be skipped from removal or removed regardless of the PRs.
* - add check sysvipcdinoex2006-12-041-0/+2
| | | | happens when security.jail.sysvipc_allowed=0
* Work around the fact that 5-exp, these days, is really another 6-exp,linimon2006-11-251-0/+4
| | | | except in disguise.
* Replace a simple call to localtime() to strftime() to include theerwin2006-11-231-1/+2
| | | | timezone.
* eik@ has been inactive for quite some time and as I am runningerwin2006-11-234-3/+6
| | | | | these scripts on the cluster, transfer maintainership of these to myself.
* Include a time when the script was started in the mail templates.erwin2006-11-231-5/+16
| | | | | | | This script can sometimes take several hours to run on builder, and thus leading to confusing of why it still reports an error that was fixed in cvs some time ago. Including the time when the ports tree was updated should reduce some of this confusion.
* Add deprecated reason to the output of -Fvd2006-11-021-2/+6
|
* * Create custom commit message template and populate it with the portsvd2006-11-011-16/+88
| | | | | | | | | | | to be removed, possibly with the expiration date and deprecated reason * If port is not marked for expiration than put "Removed" in ports/MOVED entry instead of "Has expired" * Implement -a option to remove all expired ports * Ask if the cvs diff output should be recreated/reviewed again thus giving the committer a chance to edit files by hand and view diff results afterwards * Cosmetic changes
* When checking for dependencies of foocategory/barport also find constructsvd2006-10-051-1/+1
| | | | | like .../barport/something e.g. expect either space, slash or eol after port's name
* Since we already have the data from the grep, go ahead and make the 'latestlinimon2006-09-231-5/+4
| | | | | log' cell a link to it. My weak awk/sh skills are evident here, but it does the job.
* Add links to cvsdone and INDEX files. Add some whitespace.linimon2006-09-231-2/+17
|
* When checking dependencies also look for portname references in Mk/*.mkvd2006-09-141-5/+5
| | | | in addition to */*/Makefile* and ignore references in comments
* * Back up duds to duds.old before, not after we create the new onekris2006-09-141-59/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add more XXX comments for future work * Only record a cvsdone timestamp if we updated cvs * When building with -trybroken, it's safe (and desirable) to run the prunefailure script * Reorganise a few things for better parallelism * Instead of keeping a duplicate copy of the previous logs and errors under bak/, just store a symlink to the archival location * When doing an incremental build, also cycle out the old logs to avoid broken links on the website (the logs from the previous build are removed until the packages are rebuilt). Use cpio to create hardlinked copies of the previous logs. XXX when these are bzipped by cron to save space the links will be broken and it might actually take more space. * Don't bother bunzipping old logs, now that the processlogs scripts can handle it. This was a waste of time anyway since they'd all be rebzipped by the next nightly cron job. * When the build is complete, stash a copy of the restricted ports in bak/restricted/ before deleting them from packages/, and restore from here when doing an incremental build to avoid needlessly rebuilding them each time.
* * When calculating the IP address to use for the jail, don't usekris2006-09-141-4/+5
| | | | | | | 127.0.0.0 or 127.0.0.1 * Don't use chroot to mount the linprocfs, or it won't be visible within the jail.
* * Support bzipped logfileskris2006-09-141-122/+127
| | | | | * More robust extraction of the header values (copes with the addition of the build uname to the header)
* * If .force exists, rebuild index unconditionallykris2006-09-141-5/+6
| | | | | | | * Support .log.bz2 files * Clarify that BROKEN ports are not frequently rebuilt so probably won't appear
* Correct typo in previous commitkris2006-09-141-2/+2
|
* * Drop alpha supportkris2006-09-141-6/+7
| | | | | | | | | | * Increase sparc64 build timeout to 24 hours (we have so few build machines that we cannot afford to tie them up for longer) * Increase other arch build timeout to 100 hours (hello openoffice!) * If we successfully build a formerly broken package, touch errors/.force which will kick off a rebuild of the html files
* * Retire 4-exp builds and add support for 6-exp.kris2006-09-141-11/+12
| | | | | | | | | * Use a generation number for the bindist tarballs, with compatibility symlink. Eventually we'll use this to avoid building in a "stale" chroot (i.e. populated by old world). * Don't bother running ldconfig on i386, it is evidently not needed since the other arches work fine without it
* * Silence gconf spam from the mtree checks (for now)kris2006-09-141-4/+5
| | | | | | | | * Don't try and mount/umount procfs, it won't work when we build inside a jail. * Report the uname -mr of the build environment, to ease confusion of people reading the error logs by mail.
* * Use SRCBASE for source locationkris2006-09-141-6/+12
| | | | | * The 5.x variant for populating /etc does not work on 6.x and above, and vice versa. Conditionalize old and new methods.
* Go back to force unmounting because not doing so was too unreliable.kris2006-09-141-2/+2
|
* Silence the clean-chroot invocationkris2006-09-141-1/+1
|
* Remove grep's -w switch from the dependency check since it misses somevd2006-09-121-1/+1
| | | | | dependencies. This needs some more polishing but for now it is better to emit false positives than skip dependencies.
* Get rid of multiple false positives for "arch" due to compiler warning;linimon2006-08-211-3/+1
| | | | fix true positive for "runaway_process".
* Catch a few more new cases of linker error; reclassify most of the linkerlinimon2006-08-191-0/+4
| | | | | errors seen on -CURRENT as "compat6x" e.g. "someone needs to create a compat6x port and then make it a dependency here."
* Add support for DESTDIR part I.erwin2006-08-041-10/+40
| | | | | | | | | | | | | | This commit should largele be a NOOP as it only adds support for DESTDIR undefined. This does allow us to start testing ports with DESTDIR set, but this is as of yet not supported. Although this has been extensively tested on pointyhat, this is a very intrusive change and some cases may have been overlooked. Please contact Gabor and me if you find any. PR: 100555 Submitted by: gabor Sponsored by: Google Summer of Code 2006
* Catch another case of 'disk full'.linimon2006-08-011-1/+1
|
* Change locking from per architecture to per architecture and branch.erwin2006-07-301-1/+2
| | | | | Especialy i386 has enough horsepower to run two branches simultaniously. This change has been on pointyhat for some time.
* Add an option to the addport script: -b requests that any files inshaun2006-07-141-3/+22
| | | | | | | | ${FILESDIR} which look like patches be treated as binary files. This prevents RCS tags in patch fragments causing a problem for CVS. Approved by: garga (maintainer), ahze (mentor, implicit)
* Add the cvs checkout date so people can see how long the runs take; addlinimon2006-07-131-3/+20
| | | | a comment about openoffice.org*; generalize a bit.
* Make it a little more robust.linimon2006-07-101-1/+1
|
* Convenience script to tersely display all currently running package builds.linimon2006-07-101-0/+11
|
* 'portcomment' appears to be obsolete.linimon2006-07-091-2/+0
|
* Fix the disk_full rule.linimon2006-07-091-1/+1
|
* Add the -exp builds down at the end. Probably only of interest to portmgr,linimon2006-07-081-11/+32
| | | | | so do them separately in order not to break up the ease of browsing the most important stats, above.
* White space adjustment after refactoring.linimon2006-07-081-21/+21
|
* Refactor. No code change (yet).linimon2006-07-081-11/+31
|
* Add some color to the display.linimon2006-07-061-17/+30
|
* Fix CVS tag.linimon2006-07-061-1/+1
|
* Refactor. No code changes yet.linimon2006-07-061-16/+16
|
* Add a column for last-date stamp and clean up the formatting a bit.linimon2006-07-061-7/+22
|
* Escape characters (like +) that may appear in ports' names andvd2006-07-051-2/+13
| | | | break sed -E regular expressions
* When automatic commit log is generated, change @ to " at " on submitter emailgarga2006-07-051-0/+1
| | | | to prevent spam
* Handle the case where the size of INDEX is zero (e.g. during new builds).linimon2006-06-291-1/+5
|
* Fix edit-o.linimon2006-06-281-1/+1
|
* Creates an HTML file with tables for each architecture which summarize thelinimon2006-06-271-0/+123
| | | | | number of packages built vs. packages that failed, along with some other related information.
* - Remove false positives for 'makefile' (and possibly a few others).linimon2006-06-251-57/+23
| | | | | | | | | | | | | | | | | | | | | | | - Remove obsolete explanations which are no longer seen, for speed: ELF, MOTIF, MOTIFLIB, X_manpage, awk, bison, ffs_conflict, forbidden, getopt, getopt.h, imake, lc_r, malloc.h, pod2man, sed, stl, soundcard.h, texinfo, union_wait, values.h - Add more cases to: arch, bad_c++, compiler_error, depend_object, install_error, linker_error, mtree, perl5 These changes reduce many dozens of false positives; add a few dozen true positives; and for certain directories, improve the speed about 10% (a few drop by 15%). It turns out that the performance issues are mainly due to the multiple greps. If performance is an issue we need to go back to the moderately- unreadable, everything-on-one-line paradigm. Before that happens, I would like to experiment with some refactoring, so that the patterns are built up in the shell line-by-line, so you could still be able to read it. Tested on: pointyhat Hat: portmgr
* MFP4:erwin2006-06-231-1/+1
| | | | | Actually use ${SCP_DEST_TMP} to fix the case where SCP_DEST_TMP != /tmp/
* MFP4:erwin2006-06-191-6/+24
| | | | | | When copying INDEX to the server, copy it first to a staging area and first then to the real location. The copying can take long enough for users to get a truncated file when downloading during the upload.
* - Change addport script to do the job locally instead connect to freefall everygarga2006-05-162-18/+39
| | | | | | | | | | time to add a new module. If you want to still use old way, just use "-M freefall.FreeBSD.org" option - Take addport maintainership - When modulesupdate fail, ask user to retry - Change modulesupdate to work fine with addport Approved by: will (maintainer)
* Add support for specifying ports to be removed as directories on thevd2006-05-021-1/+27
| | | | | | | | filesystem (either absolute or relative) in addition to `category/port'. PR: ports/96649 Submitted by: Cheng-Lung Sung <clsung@freebsd.org> Reworked by: vd
* Improve dependency check:vd2006-05-021-2/+3
| | | | | | When removing category/port - look if other ports' Makefiles contain `/port' rather than `category/port', since the later misses things like `${.CURDIR}/../port'
* Add support for removing multiple ports which depend on each othervd2006-04-171-36/+48
|
* Explicitly mount the NFS images ro, instead of relying on the server tokris2006-04-161-1/+1
| | | | | export it ro. This saves thrashing the server if we accidentally try to rm -rf a NFS mount (e.g. because a chroot cleanup failed).
* Don't bother archiving the ports tree, we don't use it for anything so itkris2006-04-161-1/+1
| | | | just takes up time.
* Quote ${1} so we don't get ``[: =: unexpected operator'' if the scriptvd2006-04-121-1/+1
| | | | is started without arguments.
* Add the script I use to remove ports so everyonevd2006-04-101-0/+315
| | | | can benefit (or suffer :-) from it.
* Mount linprocfs on amd64 for the convenience of jdk builds.kris2006-04-031-1/+1
|
* The squid rc.d script no longer has the .sh suffixkris2006-04-032-4/+4
|
* Do not preserve the previous package sets in bak/. This is taking upkris2006-04-031-2/+2
| | | | too much space on pointyhat, and it's also very slow.
* Clean up linprocfs on amd64 as well.kris2006-04-031-1/+1
|
* jdk is buildable on amd64, so we need to also mount linprocfs there.kris2006-04-031-2/+2
|
* This script preserves packages that will be removed by the restricted.shkris2006-04-031-0/+11
| | | | | | | | script, i.e. so they can be moved back into place before start the next incremental build so they won't be needlessly rebuilt every time (jdk, I'm looking at you). It is a bit of a hack since it relies on assumptions about the structure of that shell script, but for now it's the best we can do.
* Temporarily back out the pre.log change, it doesn't work properly.kris2006-03-131-6/+6
| | | | Approved by: portmgr (implicit)
* Mount the md image async here tookris2006-03-131-1/+1
| | | | Approved by: portmgr (implicit)
* Add some more directories temporarily until these ports can be made tokris2006-03-131-4/+8
| | | | | | clean themselves up properly. Approved by: portmgr (implicit)
* Add a mechanism for reporting client machine error conditions back to thekris2006-02-151-1/+14
| | | | | | | | | | | server. Error conditions are flagged by other processes by creating a named dotfile in ${scratchdir}. If these files are found, report the error status instead of the number of running jobs. Currently report "ERR" for all error conditions; I will probably change this to a per-condition message. Currently only "squid not running" and "disk space low" conditions are reported.
* Trim some unused cruftkris2006-02-151-69/+12
| | | | | | | If the package copy fails, bail out immediately instead of later on when we try to pkg_add it. Also trap signals and bail out. Both conditions will cause a retry of the package build.
* Save the output of portbuild run on the client to a temporary log file.kris2006-02-151-1/+11
| | | | | | | | | | If portbuild bailed out unexpectedly, mail the log to ${mailto}. Add some XXX comments about improving robustness of this script. Sleep for 2 minutes before retrying builds, to avoid spamming ${mailto} with a high rate of failure logs. In future we might be smarter about attempting to automatically correct common failure modes.
* Add some sanity checking of the build environment:kris2006-02-151-1/+30
| | | | | | | | | | | * Test whether squid is running. If not, try to kick off the rc script in the background in case it can be restarted cleanly. * Test for at least 100MB of free space on the scratch partition. If either condition fails, set an exception flag and bail out. This will be reported back to the server via reportload.
* * Don't try to pkg_delete packages that are not installed (the usualkris2006-02-151-6/+9
| | | | | | | | | | cause is because it was specified in the list twice) * Don't panic when the list of packages to delete becomes empty * When unexpected filesystem changes are detected, bail immediately instead of proceeding and hiding the error in the middle of the log
* Add support for 6-exp builds (FreeBSD 6.x is our new reference platform)kris2006-02-154-4/+14
|
* * Switch back to using pkg_delete -f; we are not yet ready to dealkris2006-02-141-25/+26
| | | | | | | | | with all the errors from broken pkg_delete scripts * As threatened in previous commit, move the pristine mtree spec generation to phase 1, and avoid having to delete and re-add the FETCH_DEPENDS. We still have to keep them installed until after 'make extract' though
* * In kill_procs(), pass in the directory and mountpoint as separatekris2006-02-111-2/+8
| | | | | | | | | | | arguments (cosmetic) * Detect if a chroot was used to run a jailed build, and first attempt to gracefully shut it down by killing everything within using pgrep(1) This has a much higher chance of succeeding that relying on fstat to identify processes that might interfere with our attempts to clean up mountpoints, which is fragile (libkvm-dependent), and inherently unreliable at best.
* Support for building ports in a jail instead of a chroot. If use_jail=1kris2006-02-111-9/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in portbuild.conf (or per-machine .conf), then construct a 127.0.0.0/8 IP address based on the build directory ID (i.e. unique for each build instance). This is bound to the lo0 interface for the duration of the 'phase 2' build. We cannot build 'phase 1' in a jail since 'make fetch' doesn't always work through a proxy (e.g. squid sometimes mangles files fetched through FTP, I think by performing CR/LF translation in FTP ASCII mode). Pass in the HTTP_PROXY variable to the jail, if set. This allows FTP/HTTP access from within the jail if the proxy is suitably configured (some ports legitimately need to fetch additional files during the build, e.g. if they have a BUILD_DEPENDS=...:configure target that needs to fetch additional distfiles). Not all ports can be built in jails (most notably the linux_base ports since they want to mount/umount linprocfs), so we will need to come up with a way to deal with this. Some ports require SYSV IPC, so security.jail.sysvipc_allowed=1 might be required. Some other ports attempt to perform DNS lookups, ping, or outbound TCP connections during the build. When it works, this provides better compartmentalization of package builds, e.g. easier termination of builds without the possibility of daemonized processes staying active; no possibility of accidental interference between jails, etc. It also allows for admin monitoring using jls(1).
* * Use ${mailto} portbuild.conf variable instead of hard-coded list [1]kris2006-02-111-4/+7
| | | | | | | | * Remove old logs and possible compressed logs before attempting the build Requested by: lofi [1] Submitted by: linimon [1] No more accidental portbuild spam: kris and krion [1]
* Use ${mailto} portbuild.conf variable instead of hard-coded listkris2006-02-111-1/+1
| | | | | | Requested by: lofi Submitted by: linimon No more accidental portbuild spam: kris and krion
* Checkpoint a major round of changes:kris2006-02-111-64/+201
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Only keep distfiles if the port passes 'make fetch', so we don't accidentally keep files with invalid checksums * Use cleanup() instead of directly exiting in some error conditions * When cleanup() is called indicating an unexpected error (possibly leaving the filesystem in an inconsistent state), mark the chroot as dirty so it will not be reused by another build * Remove packages in dependency order instead of with pkg_delete -f in possibly incorrect order. This paves the way for focusing on errors generated by pkg_delete (e.g. @dirrm that should be @dirrmtry) in the future. [1] * Detect when packages were left behind because they were still in use by other packages, indicating an incorrect or incomplete port dependency list * Partial support for ccache builds (not yet complete) * Support non-standard LOCALBASE/X11BASE settings * Delete FETCH_DEPENDS after the 'make fetch' stage. We have to add them again before 'make extract' since, due to a lack of a 'fetch cookie', 'make extract' actually *always* runs 'make fetch' again, even when distfiles have already been fetched. We need to delete them in order to: * Record an mtree spec of the 'pristine' filesystem state, for later comparison. # XXX Perhaps this can be done in stage 1 before the # 'make fetch', removing the need to delete-and-readd. * Also record an mtree spec of the filesystem state prior to the build phase. Compare this to the state of the filesystem immediately before running the install phase, to detect files that were inappropriately installed during the build phase. Doing so is a fatal error. * Prior to installing, try to run a 'regression-test' port makefile target, if it exists. This allows ports to hook their internal regression suites into the package build. This needs further infrastructure support, e.g. a default NOP target in bsd.port.mk. For now this is run with 'make -k', so regression failures will not yet actually cause package build failures. * Separate the 'make install' from 'make package' phases rather than let the latter implicitly do the install. * After the newly packaged port has been deleted, compare the state of the filesystem to the state before 'make install'. * After removing BUILD and RUN dependencies, compare the filesystem state to the pristine state before the start of the build. This also detects package dependencies that did not clean themselves up properly when deinstalling. It also detects dependencies that were 'missing' from the port INDEX: these were not pkg_added into place, so the package build had to compile them from scratch (a big waste of time and effort), so this is now also a fatal error. PR: ports/85746 (inspired by) [1] Submitted by: Boris B. Samorodov <bsam@ipt.ru> [1]
* Flip from noclean to cleankris2006-02-041-3/+3
|
* Don't bother chowning the client /var/portbuild/, it just takes too longkris2006-02-041-3/+0
| | | | and this doesn't happen in practise unless you screwed up manually.
* Use clean-chroot with clean=2 to remove the chroot instead of duplicatingkris2006-02-041-5/+1
| | | | some of the code
* Flip "noclean" to "clean"kris2006-02-041-3/+3
|
* Instead of passing in a "noclean" boolean, pass in a "clean" level:kris2006-02-041-4/+11
| | | | | | 0 = don't clean chroot 1 = sanitize commonly-modified directories 2 = remove entire chroot
* Also look for chroots that are marked 'dirty' (previous build left chrootkris2006-02-041-1/+5
| | | | in unclean state) and run clean-chroot on them to remove them.
* Don't try to remove the lock file now that we use lockf to manage itkris2006-02-041-2/+0
| | | | Noticed by: linimon
* Fix bogus comments. No code change.linimon2006-02-021-5/+5
|
generated by cgit (git 2.34.1) at 2024-11-08 12:08:57 +0800