aboutsummaryrefslogtreecommitdiffstats
path: root/Tools
Commit message (Collapse)AuthorAgeFilesLines
* Add a mechanism for reporting client machine error conditions back to thekris2006-02-151-1/+14
| | | | | | | | | | | server. Error conditions are flagged by other processes by creating a named dotfile in ${scratchdir}. If these files are found, report the error status instead of the number of running jobs. Currently report "ERR" for all error conditions; I will probably change this to a per-condition message. Currently only "squid not running" and "disk space low" conditions are reported.
* Trim some unused cruftkris2006-02-151-69/+12
| | | | | | | If the package copy fails, bail out immediately instead of later on when we try to pkg_add it. Also trap signals and bail out. Both conditions will cause a retry of the package build.
* Save the output of portbuild run on the client to a temporary log file.kris2006-02-151-1/+11
| | | | | | | | | | If portbuild bailed out unexpectedly, mail the log to ${mailto}. Add some XXX comments about improving robustness of this script. Sleep for 2 minutes before retrying builds, to avoid spamming ${mailto} with a high rate of failure logs. In future we might be smarter about attempting to automatically correct common failure modes.
* Add some sanity checking of the build environment:kris2006-02-151-1/+30
| | | | | | | | | | | * Test whether squid is running. If not, try to kick off the rc script in the background in case it can be restarted cleanly. * Test for at least 100MB of free space on the scratch partition. If either condition fails, set an exception flag and bail out. This will be reported back to the server via reportload.
* * Don't try to pkg_delete packages that are not installed (the usualkris2006-02-151-6/+9
| | | | | | | | | | cause is because it was specified in the list twice) * Don't panic when the list of packages to delete becomes empty * When unexpected filesystem changes are detected, bail immediately instead of proceeding and hiding the error in the middle of the log
* Add support for 6-exp builds (FreeBSD 6.x is our new reference platform)kris2006-02-154-4/+14
|
* * Switch back to using pkg_delete -f; we are not yet ready to dealkris2006-02-141-25/+26
| | | | | | | | | with all the errors from broken pkg_delete scripts * As threatened in previous commit, move the pristine mtree spec generation to phase 1, and avoid having to delete and re-add the FETCH_DEPENDS. We still have to keep them installed until after 'make extract' though
* * In kill_procs(), pass in the directory and mountpoint as separatekris2006-02-111-2/+8
| | | | | | | | | | | arguments (cosmetic) * Detect if a chroot was used to run a jailed build, and first attempt to gracefully shut it down by killing everything within using pgrep(1) This has a much higher chance of succeeding that relying on fstat to identify processes that might interfere with our attempts to clean up mountpoints, which is fragile (libkvm-dependent), and inherently unreliable at best.
* Support for building ports in a jail instead of a chroot. If use_jail=1kris2006-02-111-9/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | in portbuild.conf (or per-machine .conf), then construct a 127.0.0.0/8 IP address based on the build directory ID (i.e. unique for each build instance). This is bound to the lo0 interface for the duration of the 'phase 2' build. We cannot build 'phase 1' in a jail since 'make fetch' doesn't always work through a proxy (e.g. squid sometimes mangles files fetched through FTP, I think by performing CR/LF translation in FTP ASCII mode). Pass in the HTTP_PROXY variable to the jail, if set. This allows FTP/HTTP access from within the jail if the proxy is suitably configured (some ports legitimately need to fetch additional files during the build, e.g. if they have a BUILD_DEPENDS=...:configure target that needs to fetch additional distfiles). Not all ports can be built in jails (most notably the linux_base ports since they want to mount/umount linprocfs), so we will need to come up with a way to deal with this. Some ports require SYSV IPC, so security.jail.sysvipc_allowed=1 might be required. Some other ports attempt to perform DNS lookups, ping, or outbound TCP connections during the build. When it works, this provides better compartmentalization of package builds, e.g. easier termination of builds without the possibility of daemonized processes staying active; no possibility of accidental interference between jails, etc. It also allows for admin monitoring using jls(1).
* * Use ${mailto} portbuild.conf variable instead of hard-coded list [1]kris2006-02-111-4/+7
| | | | | | | | * Remove old logs and possible compressed logs before attempting the build Requested by: lofi [1] Submitted by: linimon [1] No more accidental portbuild spam: kris and krion [1]
* Use ${mailto} portbuild.conf variable instead of hard-coded listkris2006-02-111-1/+1
| | | | | | Requested by: lofi Submitted by: linimon No more accidental portbuild spam: kris and krion
* Checkpoint a major round of changes:kris2006-02-111-64/+201
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Only keep distfiles if the port passes 'make fetch', so we don't accidentally keep files with invalid checksums * Use cleanup() instead of directly exiting in some error conditions * When cleanup() is called indicating an unexpected error (possibly leaving the filesystem in an inconsistent state), mark the chroot as dirty so it will not be reused by another build * Remove packages in dependency order instead of with pkg_delete -f in possibly incorrect order. This paves the way for focusing on errors generated by pkg_delete (e.g. @dirrm that should be @dirrmtry) in the future. [1] * Detect when packages were left behind because they were still in use by other packages, indicating an incorrect or incomplete port dependency list * Partial support for ccache builds (not yet complete) * Support non-standard LOCALBASE/X11BASE settings * Delete FETCH_DEPENDS after the 'make fetch' stage. We have to add them again before 'make extract' since, due to a lack of a 'fetch cookie', 'make extract' actually *always* runs 'make fetch' again, even when distfiles have already been fetched. We need to delete them in order to: * Record an mtree spec of the 'pristine' filesystem state, for later comparison. # XXX Perhaps this can be done in stage 1 before the # 'make fetch', removing the need to delete-and-readd. * Also record an mtree spec of the filesystem state prior to the build phase. Compare this to the state of the filesystem immediately before running the install phase, to detect files that were inappropriately installed during the build phase. Doing so is a fatal error. * Prior to installing, try to run a 'regression-test' port makefile target, if it exists. This allows ports to hook their internal regression suites into the package build. This needs further infrastructure support, e.g. a default NOP target in bsd.port.mk. For now this is run with 'make -k', so regression failures will not yet actually cause package build failures. * Separate the 'make install' from 'make package' phases rather than let the latter implicitly do the install. * After the newly packaged port has been deleted, compare the state of the filesystem to the state before 'make install'. * After removing BUILD and RUN dependencies, compare the filesystem state to the pristine state before the start of the build. This also detects package dependencies that did not clean themselves up properly when deinstalling. It also detects dependencies that were 'missing' from the port INDEX: these were not pkg_added into place, so the package build had to compile them from scratch (a big waste of time and effort), so this is now also a fatal error. PR: ports/85746 (inspired by) [1] Submitted by: Boris B. Samorodov <bsam@ipt.ru> [1]
* Flip from noclean to cleankris2006-02-041-3/+3
|
* Don't bother chowning the client /var/portbuild/, it just takes too longkris2006-02-041-3/+0
| | | | and this doesn't happen in practise unless you screwed up manually.
* Use clean-chroot with clean=2 to remove the chroot instead of duplicatingkris2006-02-041-5/+1
| | | | some of the code
* Flip "noclean" to "clean"kris2006-02-041-3/+3
|
* Instead of passing in a "noclean" boolean, pass in a "clean" level:kris2006-02-041-4/+11
| | | | | | 0 = don't clean chroot 1 = sanitize commonly-modified directories 2 = remove entire chroot
* Also look for chroots that are marked 'dirty' (previous build left chrootkris2006-02-041-1/+5
| | | | in unclean state) and run clean-chroot on them to remove them.
* Don't try to remove the lock file now that we use lockf to manage itkris2006-02-041-2/+0
| | | | Noticed by: linimon
* Fix bogus comments. No code change.linimon2006-02-021-5/+5
|
* Correctly set SRCBASE; the version inherited from buildenv is wrong forkris2006-01-281-0/+1
| | | | the client chroots.
* Don't try to rsync docs, we no longer need to distribute themkris2006-01-281-1/+0
|
* Update the list of quickportskris2006-01-281-1/+1
|
* Warn if there are duplicate MD5/SHA256/SIZE statements for a single fileedwin2006-01-251-3/+33
|
* Small tool to check the necessary lines in the distinfo files, basededwin2006-01-241-0/+105
| | | | | | | | | | | | | | | on the contents of the distinfo files. # # Small tool to find distinfo with missing MD5/SHA256/SIZE statements, # based on the assumption that if there is one of the MD5/SHA256/SIZE # statements, then there should be all of them (except for SIZE # when MD5/SHA256 is set to IGNORE). # # Usage: distinfochecker [-v] [-d directory] # -v - verbose (print) # -d - use directory instead of /usr/ports #
* Support 7.x INDEX buildskris2006-01-231-16/+37
| | | | | Add a few more local configuration variables Minor cleanups
* Don't bother mounting doc tree, it's not needed by any portskris2006-01-221-12/+8
| | | | | | Use LOCALBASE/X11BASE values from buildenv xorg uses the stock BSD.x11-4.dist now.
* Remove -nodoccvs option, it turns out mounting the doc tree is notkris2006-01-221-14/+1
| | | | required by any ports
* Also clean up linprocfs mountskris2006-01-221-0/+1
|
* Pull in buildenv for LOCALBASE/X11BASE locationskris2006-01-221-2/+4
| | | | Don't bother cleaning up doc mounts, they are not needed.
* Use LOCALBASE and X11BASE from buildenvkris2006-01-221-1/+4
|
* Allow for the possibility of overriding LOCALBASE and X11BASEkris2006-01-221-0/+2
|
* Exit with the correct error code when a make target failskris2006-01-151-3/+3
| | | | | PR: ports/64930 Submitted by: Damir Kiramov <damirycha@damirycha.net.ru>
* Back out last commit, it was the wrong filekris2006-01-151-17/+15
|
* Exit with the correct error code when a make target failskris2006-01-151-15/+17
| | | | | PR: ports/64930 Submitted by: Damir Kiramov <damirycha@damirycha.net.ru>
* Add portsvar.sh schell script into Tools/scripts.daichi2006-01-111-0/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | portsvar.sh - print out variable defined through bsd.port.mk example: % portsvar.sh PORTSDIR OSVERSION PORTSDIR = /usr/ports OSVERSION = 600034 % portsvar.sh -e "PORTS[^_]" PORTSDIR = /usr/ports PORTSSUPFILE = /usr/share/examples/cvsup/ports-supfile % portsvar.sh USE_RUBY=yes RUBY_VER RUBY_ARCH RUBY_VER = 1.8 RUBY_ARCH = i386-freebsd6 % portsvar.sh -w ruby RUBY_VER RUBY_ARCH RUBY_VER = 1.8 RUBY_ARCH = i386-freebsd6 % portsvar.sh -w kde -e KDE_ KDE_MAINTAINER = will@FreeBSD.org KDE_ORIGVER = ${KDE_VERSION} KDE_PREFIX = ${LOCALBASE} KDE_VERSION = 3.5.0 MASTER_SITE_KDE_kde = ${kmaster:S@%SUBDIR%/@@g} % portsvar.sh -a list all variables through bsd.port.mk % portsvar.sh -a -w kde list all variables through bsd.port.mk including kde portsvar.sh is easy tool to check out the variables used in /usr/ports/Mk/bsd.*.mk.
* This script may be run on a client machine to force it to flush its squidkris2005-12-111-0/+15
| | | | | cache. This is useful if the client is holding a bad or stale cached copy of a package that is causing problems for package builds.
* Bump the file size limit to 300MB now that openoffice distfiles exceedkris2005-12-111-5/+5
| | | | 256MB. Clarify a comment.
* Shuffle the order of a cdkris2005-12-111-2/+1
|
* Remove 0-sized entries to avoid misbehaviourkris2005-12-111-0/+4
|
* Don't use a lockfile, the script will be called with lockf -t0 instead tokris2005-12-112-16/+0
| | | | avoid problems with stale lockfiles after a reboot.
* Don't use a lockfile, the script will be called with lockf -t0 insteadkris2005-12-111-6/+0
| | | | to avoid problems with stale lockfiles after reboots.
* Update the list of quickportskris2005-12-111-1/+1
|
* Don't set -nocopy when -norsync is set, otherwise the setupnode scriptkris2005-12-111-1/+1
| | | | doesn't completely prep the client.
* Don't use a lock file, the script will be called with lockf -t 0 instead.kris2005-12-111-6/+0
| | | | This avoids problems from stale lockfiles after reboots.
* Mount the md image async since this gives much better performancekris2005-12-111-1/+1
|
* Utility: A tool to identify similar pkg-descr filesedwin2005-11-261-0/+24
| | | | | | | | | | The ports tree contains many tools with almost-similar names and no helpful pkg-descr difference. The goal of pkg-descr would be to aid the user in his choice of port version. Ergo, similar pkg-descr files are bad. PR: ports/89432 Submitted by: Jonathan <afarsec@012.net.il>
* Remove, don't comment out, unused variableskris2005-10-111-2/+0
|
* Remove unused variables. Don't bother protecting against negativekris2005-10-111-4/+2
| | | | queue lengths since that doesn't arise.
* Update lockf to lockf -k in example usagekris2005-10-111-1/+1
|
* Update the lockf command to add -k in example usagekris2005-10-111-1/+1
|
* When -norsync is set, don't bother to run md5 over the bindist.tarkris2005-10-111-3/+9
| | | | | | | | | | and copy it to the client. This has two benefits: 1) Avoids spamming the master with dozens of md5 processes when dosetupnode is spawned for all client machines at ocne 2) Avoids silly copy attempts on disconnected nodes for which the file is copied to itself
* Change lockf to lockf -k in a commented-out command, just in case it'skris2005-10-111-1/+1
| | | | ever uncommented.
* Add support for use_md_swapkris2005-10-111-0/+5
|
* Add support for use_md_swap. When this variable is set in akris2005-10-111-4/+18
| | | | | | | | | | | | | | | | | | | | | | portbuild.conf, builds will each be done in a separate swap-backed md. This dramatically improves build performance since * Every transaction is not written to disk, so disk bandwidth is not a bottleneck * Multiple builds do not contend with each other for the same set of filesystem locks and other per-device resources The size of the md devices is controlled by the md_size variable. '2g' seems to be a good size. Currently we mdconfig -u each device after each port build, since otherwise dirty blocks accumulate and the md eventually uses a full amount of backing store (2g in the above example). This is a problem if there is unsufficient swap backing to accomodate them all. XXX This should be made configurable to avoid the performance penalty on systems that do have enough swap backing
* No need to hold the queue lock while polling client loadkris2005-10-111-1/+1
|
* * Use lockf -k instead of lockf. If there is ever contention from >1kris2005-10-111-5/+5
| | | | process, you must use -k or lockf can give out two locks at once
* * Add support for use_md_swapkris2005-10-111-12/+22
|
* * Unmount linprocfs around pkg_adding dependencies, to try and workkris2005-10-111-3/+41
| | | | | | | | | | | | | | | around silly failures in some ports when it is present * Add a footer to completed builds that is recognised by pdispatch to retry truncated builds * If ALWAYS_KEEP_DISTFILES is not present in the environment or port makefile, then clean up DISTDIR after the build finishes, to prevent collection of distfiles for this port. * Finish flipping the switch on -noplistcheck - this is activated by passing in the NOPLISTCHECK environment variable instead of using PLISTCHECK in the opposite case
* * Finish flipping the switch on -noplistcheck - this is activated bykris2005-10-111-11/+6
| | | | | | | | | passing in the NOPLISTCHECK environment variable instead of using PLISTCHECK in the opposite case * Remove the unused -nodummy function * Pass in ALWAYS_KEEP_DISTFILES when -distfiles is set
* * Remove support for the unused -nodummy functionkris2005-10-111-65/+66
| | | | | | | | | | | | | * Finish flipping the switch on -noplistcheck - this is activated by passing in the NOPLISTCHECK environment variable instead of using PLISTCHECK in the opposite case * Always pull in the distfiles from the client if they exist (needed for forthcoming ports tree changes to satisfy GPL license requirements) * If the build did not complete "cleanly", e.g. it was interrupted by a network outage or client machine panic, then retry it until it succeeds instead of just leaving a dirty truncated log
* * Remove unused -nodummy argumentkris2005-10-111-34/+6
| | | | | | | | | | | * Finish flipping the switch on -noplistcheck; set NOPLISTCHECK instead of PLISTCHECK in the opposite case * Clean up the distfiles/ directory when starting build, so it is not contaminated by old distfiles * Remove commented-out code to don't back up old distfiles/ directory, it's just too big to do this.
* Silence some possible errors from chflags -Rkris2005-10-111-1/+1
|
* Silence possible errors from some recursive commandskris2005-10-111-2/+2
|
* - Add a new option -L, it works like -l but uses pkg-descr to creategarga2005-10-051-17/+30
| | | | | | commit message. Approved by: will (maintainer)
* Remove some hardcoded values to hopefully stop random users from shootingkris2005-09-091-12/+27
| | | | | | their feet off in public. Suggested by: adamw
* - Handle different INDEX filename on 5.X and abovepav2005-09-051-2/+5
| | | | | PR: ports/85281 Submitted by: Vasil Dimov <vd@datamax.bg>
* Check for extra files on !i386 alsokris2005-07-251-6/+4
|
* Support FreeBSD 7.xkris2005-07-255-5/+15
|
* Drop the removed openoffice 1.0 ports from the quickports listkris2005-07-251-1/+1
|
* Correct typokris2005-07-251-1/+1
|
* Add prototypes for all the subroutines which fixes compilation withmarcus2005-07-111-9/+20
| | | | | | Perl 5.8.7. Reported by: mezz
* Fix typosvs2005-04-181-3/+3
| | | | Approved by: sobomax
* As discussed on the cvs-ports-list, make 'addport' print a warning if the ↵vs2005-04-121-0/+1
| | | | | | | | | directory name contains upper-case characters as it makes finding a port harder. PR: ports/79679 Approved by: will, portmgr(krion)
* Add a splitpatch.pl utility, a small script to convert multi-filetobez2005-04-122-0/+112
| | | | | | patches to several appropriately named single-file patches. Requested by: flz
* - Add bump_revision.pl, a small script to bump the PORTREVISION variableflz2005-04-122-1/+193
| | | | | | | | | | of ports which are depending on a port with a changed shared lib version. Edwin dedicates this script to his son Dirk, born few days ago :) Congrats (again) ! PR: ports/63915 Submitted by: edwin
* - Replace pkg-comment with COMMENT to reflect latest (*sigh*) changesflz2005-04-121-2/+2
| | | | | | | | in the ports framework. PR: ports/62767 Submitted by: Igor Pokrovsky <tiamat@comset.net> Approved by: maintainer timeout (> 8 months)
* Use WRKDIRPREFIX instead of hardcoded value.kris2005-04-121-1/+3
| | | | Clean up after we're done.
* Change rsync -l to -L for the tarballs/ directory (so that danglingkris2005-04-031-1/+1
| | | | | symlinks are not created on the target) and rsync the entire directory so that extra files are deleted.
* Build packages with WRKDIRPREFIX=/work (which does not exist bykris2005-04-031-1/+9
| | | | | | | default) instead of /tmp, since some packages incorrectly search this build path at runtime to try to load things. Requested by: nectar
* Increase the nc poll timeout to better deal with network latencies.kris2005-04-031-1/+1
|
* Flip over the -plistcheck switch to -noplistcheck since we now want tokris2005-04-031-5/+5
| | | | make it the default behaviour.
* Update the list of files and directories to filter from the plist checking.kris2005-04-031-1/+3
|
* Checkout Templates from CVS as well since ports with GNU_CONFIGURE willvs2005-03-211-0/+2
| | | | | | need them when building from 'addport -a -t'. Approved by: will
* Put $FreeBSD$ for ident(1)'ing.krion2005-03-131-0/+1
|
* Hopefully last try to fix readmes build.krion2005-03-121-1/+2
| | | | | Submitted by: Christopher Kelley <bsd at kelleycows.com> Pointyhat to: krion
* Fix problems with quoting if PORTSTOP is specified.krion2005-03-101-1/+1
|
* * Add info files to TMPPLIST, only if INFO is specified. [1]krion2005-03-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | | * Fix PATH problems during README.html generation if "make -DPORTSTOP readmes" is defined. [2] * Add support for early rcNG startup scripts. [3] * Remove duplicate USE_XLIB handling. [4] * Add check for USE_PYTHON_(BUILD|RUN), and include bsd.python.mk if both are specified. [5] * Use PRE-INSTALL target for SUB_FILES and SUB_LIST. [6] * Introduce bsd.gstreamer.mk and utilize new macros: USE_GSTREAMER/WANT_GSTREAMER. [7] PR: ports/77634 [1], ports/77592 [2], ports/77385 [3], ports/77557 [4], ports/77344 [5], ports/77211 [6], ports/77838 [7] Submitted by: krion [1] [2], Florent Thoumie <flz@xbsd.org> [3] [6], pav [4], lofi [5], ahze [7]
* * Instead of using umount -f to unmount things, first use fstat tokris2005-02-121-6/+44
| | | | | | | | | look for processes holding open references within the FS and kill them, then use regular umount. This is necessary now that devfs cannot be force-unmounted, and has the benefit that processes can't hang around holding references to files between port builds. * Preliminary work to support using ccache to accelerate builds.
* Only try to chown the portbuild directory if we are a disconnected clientkris2005-02-121-1/+1
| | | | (i.e. it is not mounted by nfs)
* Instead of using umount -f to unmount things, first use fstat to look forkris2005-02-121-1/+35
| | | | | | | processes holding open references within the FS and kill them, then use regular umount. This is necessary now that devfs cannot be force-unmounted, and has the benefit that processes can't hang around holding references to files between port builds.
* * Instead of using umount -f to unmount things, first use fstat tokris2005-02-121-7/+44
| | | | | | | | | | | look for processes holding open references within the FS and kill them, then use regular umount. This is necessary now that devfs cannot be force-unmounted, and has the benefit that processes can't hang around holding references to files between port builds. * Reduce possibility for error by testing for presence of executable ldconfig inside the chroot before attempting to run it (e.g. it may not be there if the chroot was not completely initialized)
* Update the plist checking to look for a subset of files and weed out somekris2005-02-121-1/+6
| | | | false positives.
* Document the check-latest-link and tindex scriptskris2005-01-241-1/+5
|
* Add the script I use to look for duplicate LATEST_LINK entries and tokris2005-01-241-0/+72
| | | | send nag-mails to the responsible maintainers.
* This is the script I use to build indexes for all supported ports branches,kris2005-01-241-0/+164
| | | | | | which are copied to www.freebsd.org for use by 'make fetchindex', as well as the source of build failure reports on ports@FreeBSD.org when it is broken.
* Remove traling spaces.krion2005-01-0928-52/+52
|
* Always set up the bindist-${hostname}.tar file since it's usually small.kris2005-01-071-1/+1
| | | | | | This should probably be changed to only update it if it changes. Approved by: portmgr (self)
* Unset the __MAKE_CONF variable after setting up buildenv, because wekris2005-01-071-0/+3
| | | | | | want to use the default value in the build chroot. Approved by: portmgr (self)
* Allow machines to use an alternate address for polling for job load (e.g.kris2005-01-071-1/+3
| | | | | | | if the machine is behind a NAT and is not directly addressable, we can use a localhost port forward to contact it) Approved by: portmgr (self)
* * Include portbuild.conf before portbuild.${hostname} to reset the defaultkris2005-01-071-4/+6
| | | | | | | | value of variables * Allow krion to drink from the email firehose Approved by: portmgr (self)
* * The mlist file now only has one field per line.kris2005-01-071-4/+9
| | | | | | | | * Correctly calculate job concurrency in pass two as well * In pass two run dosetupnode -rsync to avoid wasting time. Approved by: portmgr (self)
* krion wants to drink from the pointyhat email firehose as well.kris2005-01-071-1/+1
| | | | | | XXX This should be a portbuild.conf tunable Approved by: portmgr (self)
* Now that I have so many remote build machines, don't force trying tokris2005-01-071-7/+0
| | | | | | | | fetch from ftp-master and pointyhat; they'll just get timeouts. Instead, each machine is expected to set up their own MASTER_SITE_* variables in etc/make.conf via a bindist-${hostname}.tar file. Approved by: portmgr (self)
* * Support the -norsync argument to only remove the existing build chrootskris2004-12-281-2/+8
| | | | | | | | | | | | | | | | | on a disconnected client, without running the time-consuming rsyncs. This is useful when a build is interrupted and needs to be restarted. * After we have cleaned up the machine, reset the queue counter by using pollmachine -queue. This has a race condition if other builds are being dispatched to the machine (e.g. builds on another branch): getmachine can claim a directory and increment the counter, then the machine is polled and finds e.g. 0 chroots in use, and resets the counter to 0, then claim-chroot is run and the build dispatched, with the counter now off-by-one. This could be fixed by running claim-chroot with the .lock held, but this turns out to be too time-consuming. A two-level lock approach might also fix this efficiently.
* * It does in fact make sense to specify -incremental and -noindex at thekris2004-12-281-7/+7
| | | | | | | | | | same time, assuming that the admin has already built the INDEX and INDEX.old in advance. * Adapt to new method of calculating build concurrency, by summing the value of ${maxjobs} listed in every portbuild.${machine} * Support 5-exp builds
* Fix the Xvfb code to run Xvfb before the build phase if detectedkris2004-12-281-15/+25
| | | | | | | | | | | (i.e. if the package lists a dependency on the relevant package in the PACKAGE_BUILDING case). This allows packages that require an available DISPLAY to again build (with some forthcoming fixes to existing ports). Improve the reporting of detected filesystem anomalies (extra files left behind after deinstallation, changes to and removal of pre-existing files)
* Overhaul of the job scheduler. The new scheduler runs buildskris2004-12-284-29/+119
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | synchronously instead of probabilistically scheduling jobs, which means that the job load on a machine never exceeds a desired threshold, and we can preferentially use faster machines when they are available. This has a dramatic effect on package build throughput, although I don't yet have precise measurements of the performance improvements. Specifically, the changes are: * Introduce the new variable maxjobs in portbuild. This replaces the build scheduling weights previously listed in the mlist file, which now changes format to list the build machines only, ranked in order of preference for job dispatches (i.e. faster machines first). * The ${arch}/queue directory is used to list machines available for jobs (file content is the number of jobs currently running on the machine). Changes to files in this directory are serialized using lockf on the .lock file. * Claim a machine with the getmachine script, with the .lock held. This picks the machine with the fewestnumber of jobs running, which is listed highest in the mlist file in case of multiple machines with equal load. The job counter is incremented, and the file removed if the counter reaches ${maxjobs} for that machine. If all machines are busy, sleep for 15 seconds and retry. * After we have claimed a machine, we run claim-chroot on it to claim an empty chroot, as before. If the claim fails, release the job from the queue with the releasemachine script and retry after a 15 second wait. * When the build is finished, decrement the job counter with the releasemachine script, with .lock held. * The checkmachines script now exists only to poll the load averages for admin convenience (every 2 minutes), and to ping for unreachable machines. When a machine cannot be reached, remove the entry in the queue directory to stop further job dispatches to it. This needs more work to deal with reinitialization of machines after they become available again.
* Overhaul of the job scheduler. The new scheduler runs buildskris2004-12-281-26/+33
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | synchronously instead of probabilistically scheduling jobs, which means that the job load on a machine never exceeds a desired threshold, and we can preferentially use faster machines when they are available. This has a dramatic effect on package build throughput, although I don't yet have precise measurements of the performance improvements. Specifically, the changes are: * Introduce the new variable maxjobs in portbuild. This replaces the build scheduling weights previously listed in the mlist file, which now changes format to list the build machines only, ranked in order of preference for job dispatches (i.e. faster machines first). * The ${arch}/queue directory is used to list machines available for jobs (file content is the number of jobs currently running on the machine). Changes to files in this directory are serialized using lockf on the .lock file. * Claim a machine with the getmachine script, with the .lock held. This picks the machine with the fewestnumber of jobs running, which is listed highest in the mlist file in case of multiple machines with equal load. The job counter is incremented, and the file removed if the counter reaches ${maxjobs} for that machine. If all machines are busy, sleep for 15 seconds and retry. * After we have claimed a machine, we run claim-chroot on it to claim an empty chroot, as before. If the claim fails, release the job from the queue with the releasemachine script and retry after a 15 second wait. * When the build is finished, decrement the job counter with the releasemachine script, with .lock held. * The checkmachines script now exists only to poll the load averages for admin convenience (every 2 minutes), and to ping for unreachable machines. When a machine cannot be reached, remove the entry in the queue directory to stop further job dispatches to it. This needs more work to deal with reinitialization of machines after they become available again. Additional changes to this file: * Exit if passed a null package name, to avoid badness later on * Send a nag-mail if pkg-plist errors are detected in the build
* * Catch up to change in formatting of 'extra files' section in logskris2004-12-281-4/+3
| | | | * Other minor changes
* /rescue/mount_linprocfs isn't functional when called bykris2004-12-281-1/+2
| | | | | | | | | /rescue/mount -t linprocfs, so assume that the i386 build hosts have statically-built copies of the necessary binaries in /sbin, until this is fixed. Create /usr/X11R6 inside the chroot so that mtree has something to do, since this directory is otherwise orphaned.
* Support 5-exp buildskris2004-12-284-4/+14
|
* Use X_WINDOW_SYSTEM for 4.x builds too.kris2004-12-101-2/+2
|
* Belatedly catch up to XFREE86_VERSION -> X_WINDOW_SYSTEM change; applykris2004-12-101-3/+3
| | | | the BSD.x11.dist if set to xfree86-3, otherwise use BSD.x11-4.dist.
* The explanations of each build error no longer live on bento, they livelinimon2004-12-092-7/+7
| | | | on pointyhat.
* Construct permanent links to the error logs here too.kris2004-12-051-4/+4
| | | | | Submitted by: linimon Reminded by: marcus
* Copy the statically-linked binaries from /rescue on the host, insteadkris2004-12-051-4/+11
| | | | | of needing to also include the dynamic libraries that were confusing some ports. We still need libc.so.[56] for killall to work, though :(
* Be silent about attempts to unmount filesystems from within the chrootkris2004-12-051-4/+4
|
* Clean up the symlink in ${pb}/${arch}/${branch}/latest/${dir} when we findkris2004-11-251-0/+5
| | | | a port that should not be listed here
* If sudo_cmd is set, assume we need to chown the files we created onkris2004-11-251-0/+3
| | | | the client to the client_user so they can later write to them
* Rework the -incremental support slightly; this should now be robust enoughkris2004-11-251-3/+5
| | | | to use.
* Catch up to the changed format of the logs.kris2004-11-251-4/+4
| | | | | | | List the extra/removed/changed files separately, and colour-code the serious errors (files left behind outside of /usr/local and /usr/X11R^; files removed that were installed by another port, and files with changed permissions or ownership)
* * Recursively list all files detected as having been left behind afterkris2004-11-251-7/+22
| | | | | | | the port deinstall; mtree does not recurse into subdirectories it does not know about * Break out the 'files incorrectly removed' and 'files incorrectly changed' into their own sections
* * Fix INDEX build if MAINTAINER is not defined. [1]krion2004-11-191-0/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Remove USE_QT2 since it's obsolete now. [2] * Clarify comments about ARCH. [3] * Speedup 'make readmes'. Add a perl script "Tools/make_readmes" and modify bsd.port.subdir.mk to avoid recursing into individual port directories to create README.html. [4] * Fix 'make search' to allow case insensitive search on 5-x/6-x. [5] * Add the possibility to search the ports by category. [6] * Remove tk42 and tcl76 from virtual categories since they're obsolete. [7] * Introduce new variable - DISTVERSION, vendor version of the distribution, that can be set instead of PORTVERSION and is automatically converted in a conforming PORTVERSION. [8] * Use --suffix instead of -b option for patch(1) to make it compatible with BSD patch(1) [9] * Fix {WANT,WITH}_MYSQL_VER behavior, to deal with conflicting versions. [10] PR: ports/68895 [1], ports/69486 [2], ports/68539 [3], ports/70018 [4], ports/68896 [5], ports/73299 [6], ports/73570 [7], ports/67171 [8], ports/72182 [9] Submitted by: linimon [1][3], arved [2][7], cperciva [4], Matthew Seaman <m.seaman@infracaninophile.co.uk> [5], Radek Kozlowski <radek@raadradd.com> [6], eik [8], Andreas Hauser <andy-freebsd@splashground.de> [9], clement [10]
* Newline should go after the terminating punctuation (fix typo).dd2004-11-071-1/+1
| | | | Approved by: will
* Fix a couple of bugs in the extraction of this from dopackages, andkris2004-10-251-33/+20
| | | | | flatten the setupnode() function into the main body since it's only called from one place.
* Redefine the -norestr option to mean 'don't build the list ofkris2004-10-251-78/+73
| | | | | | | | | | restricted ports' instead of 'don't build any restricted ports' since the former is useful when we're not intending to publish the results of a build, but the latter is not. Move the build preprocessing (directory setup, old build rotation, etc) out from under -nobuild, so that we can set up a new build using that option.
* If the symlink under latest/ exists, then use that to find the most recentkris2004-10-251-1/+6
| | | | | error log for the port, instead of assuming it exists in the current build.
* We now also do 6.x builds in the '6' branch.kris2004-10-251-3/+3
|
* Use the absolute path to nc(1) to deal with it not being in the PATH.kris2004-10-251-3/+1
| | | | Remove useless sleep(1).
* Clean up the 'latest error log' symlink when the build is successful.kris2004-10-251-0/+4
|
* When a port build fails, create a symlink to the error log underkris2004-10-251-3/+4
| | | | | | | ${arch}/${branch}/latest/${portdir}. We will use this in the processfail script, so that the "new package build errors" webpages do not have out-of-date links but instead link to the most recent copy of the build error.
* Strip out date/time from +++/--- headers in generated patchfiles, so that theysobomax2004-10-141-11/+12
| | | | don't generate extra hunk each time patchfile is updated.
* Support ${client_user} and ${sudo_cmd} for running commands as root onkris2004-09-271-1/+5
| | | | | | | machines where we first connect to a non-root user and then increase privileges. Approved by: portmgr (self)
* * Break out setupnode into the dosetupnode script so it may be calledkris2004-09-271-37/+3
| | | | | | | | manually * Support 6.x builds Approved by: portmgr (self)
* Break out the setupnode function of dopackages into a separate script sokris2004-09-271-0/+47
| | | | | | | | | | | | | | | that it may be called by hand. Support new portbuild.conf variables client_user = user to connect to on the client (not necessarily root). This user must have write permission to the /var/portbuild tree if disconnected=1 (i.e. we're going to run rsync). rsync_gzip = set to "-z" to enable compression on low-bandwidth disconnected clients. Approved by: portmgr (self)
* * Bail out if we can't claim a chroot directory in the client (e.g. ifkris2004-09-271-13/+17
| | | | | | | | | | | | | | | ssh times out) * Support new portbuild.conf settings: client_user = user to connect to on the client (not necessarily root) sudo_cmd = If ssh'ing to a non-root user, run this command to gain root privs (set to empty string for client_user=root, or sudo for !root). Cannot require interactivity, of course. Approved by: portmgr (self)
* Refuse to "clean" / if something goes wrong upstream and we are passedkris2004-09-271-0/+5
| | | | | | the wrong path Approved by: portmgr (self)
* Pull in portbuild.$(hostname) for per-client customizationkris2004-09-271-1/+3
| | | | Approved by: portmgr (self)
* Support 6.x buildskris2004-09-272-1/+5
| | | | Approved by: portmgr (self)
* * Support 6.x buildskris2004-09-271-2/+12
| | | | | | | * Default 4.x to XFree86 4.x, 5.x and above to x.org * Don't pick up local customizations from OPTIONS Approved by: portmgr (self)
* Use a consistent PATH in both portbuild and dopackages.marcus2004-08-281-1/+1
|
* Avoid possible GNATS timeouts by using query-pr.real direclty.krion2004-07-281-1/+1
|
* Clean up the specified chroot (umount filesystems, and scrub thekris2004-07-141-0/+36
| | | | directories likely to contain changed files).
* Atomically claim an unused chroot directory. The previous algorithmkris2004-07-141-0/+52
| | | | | was not atomic and could have ended up with two port builds trying to use the same chroot. Report the claimed directory back to the caller.
* Bump up the cleanup timeout to directories that have been in use forkris2004-07-141-3/+3
| | | | >48 hours, to cope with monster ports like openoffice and atlas.
* Back out the attempt to list the error type in the failure summary;kris2004-07-141-14/+1
| | | | | | | because this file is a chronological history of port builds that have failed, the files listed may not be present in the current set of error logs, and we currently have no easy way to find the most recent failure log to use instead.
* Generate permanent URLs in the error reports. The directories likekris2004-07-141-1/+10
| | | | | | | | i386-5-latest that are linked to from the index.html are symlinks to dated directories (e.5.`date`), so the URLs in the error reports will expire with the start of the next build when the symlink is repointed. This change makes the URLs in the error reports use the realpath of the target file, so they do not expire.
* * Add support for ${TRYBROKEN}.kris2004-07-141-6/+67
| | | | | | | | | | | | | | | | | | | | * Clients no longer have ssh access to the master, so we need to push/pull everything on the client from here. This means we need to know where the build took place so we can go in and get the files after it finishes. Introduce the claim-chroot script which atomically claims a free chroot directory on the host and returns the name. This directory is later populated by the portbuild script if it does not already contain an extracted bindist. * Use the per-node portbuild.$(hostname) config file to decide where in the filesystem to claim the chroot on the build host. * If a port failed unexpectedly (i.e. is not marked BROKEN), or if something strange happened when trying to pull in build results from a client, then send me email (XXX should be configurable). * Clean up after the build finishes and we have everything we need, by dispatching the clean-chroot script on the client.
* * Add a cleanup() function and use it so that we archive the ${WRKDIR}kris2004-07-141-68/+122
| | | | | | | | | | | | | | | | | | | if requested (".keep" file in the port directory), no matter where we fail. * Add package dependencies before the corresponding build stage (e.g. FETCH_DEPENDS before 'make fetch'), and remove them again afterwards. This allows us to catch ports that list their dependencies too early/late. * No need to check for set[ug]id files here, the security-check target in bsd.port.mk does it for us. * Exclude some more directories and files from showing up in the mtree before/after comparison, to trim down the false-positive in the pkg-plist check. * Other minor changes
* * Remove the quickports stuff, it didn't (ever?) work as designed andkris2004-07-141-112/+156
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | it's done properly^Wbetter in makeparallel * Script accepts new arguments: -nodoccvs: skip cvs update of the doc tree -trybroken: try to build BROKEN ports (off by default because the i386 cluster is fast enough now that when doing incremental builds we were spending most of the time rebuilding things we know are probably going to fail anyway. Conversely, the other clusters are slow enough that we also usually don't want to waste time on BROKEN ports). -incremental: compare the interesting fields of the new INDEX with the previous one, remove packages and log files for the old ports that have changed, and rebuild the rest. This substantially cuts down on build times since we don't rebuild ports that we know have not changed. XXX checkpoint of work-in-progress, not yet working as committed. * When setting up the nodes, read in per-node config files ("portbuild.$(hostname)") before dispatching the setupnode script on each node. For disconnected nodes (which don't mount the master via NFS), we also rsync the interesting files required by the builds (ports/src/doc trees, bindist tarballs, scripts) into place on the client. They will be mounted locally via nullfs in the build chroots. * Break out the restricted.sh generation into a makerestr script so it can be called manually as needed. * Remove the -nocvsup argument which has been unused for a long time. * For now, don't prune the list of failed ports with prunefail, since when -trybroken is not specified, every BROKEN port ends up in the duds file (so the build is skipped), and as a result we would prune almost everything from the list of failed ports. XXX prunefailure should be run conditionally on -trybroken, or I should find a way to prune in both cases. * Don't run index in the background, it was thrashing against makeduds and not saving any time by doing it concurrently. * Build with 'make quickports all' to kick off the quickports builds earlier. * Delete restricted and/or cdrom distfiles *after* post-processing the distfiles, otherwise the script doesn't remove any of them since they're not in the expected place. * Miscellaneous other minor changes and cleanups
* Parse the slightly-different format of the build logs.kris2004-07-141-1/+1
|
* Parse the slightly-different format of the build logs.kris2004-07-141-3/+3
|
* * Accept another argument, the value of the disconnected variable thatkris2004-07-141-25/+25
| | | | | | | | | | | | | | | | | | | | | | | tells us whether the node has NFS access to the master. * Also copy the bindist-$(hostname).tar file to allow local customization of the build chroots (e.g. resolv.conf and make.conf files for disconnected systems) * For disconnected hosts, we don't copy the bindist files from the master, but just set up the local directories and let the server rsync them into place later. Also set up dangling symlinks to the bindist files in the build area, which will be filled in by the server too (in the NFS case it makes sense to cache the bindist files locally to avoid extra NFS traffic, but here we know the file is local so a symlink is fine) * Remove an apparently spurious 'killall fetch' that snuck in for what were probably transient reasons. * Forcibly clean up old chroot directories since we are preparing to start another build and don't want old (possibly orphaned) builds to skew the job scheduling or use up resources.
* Allow local customization of the node using the portbuild.$(hostname) file.kris2004-07-141-0/+1
|
* Send me mail when a previously-broken port builds successfully.kris2004-07-141-2/+5
| | | | XXX Should not be a hard-coded address
* * Initial support for disconnected nodes (not on same LAN as buildkris2004-07-141-83/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | host), specified by disconnected=1 in portbuild.$(hostname) file. These do not mount via NFS, so we need to maintain a local copy of things needed by the build (like the ports/src/doc trees) on the build host, which are mounted into the chroot by read-only nullfs. These local files are maintained in the dopackages script via rsync. * Download packages via http instead of NFS. Allow fetching via a local http proxy (http_proxy variable in per-node portbuild.$(hostname) file). Caching package dependencies saves about 85% of package fetches and similar reduction in package fetch traffic by byte count. * Support a per-node tarball (bindist-$(hostname).tar) to customize the build chroots. This is used for things like local resolv.conf and make.conf files on disconnected nodes. * Make sure we don't use a chroot until it is finished extracting. * Don't set '.' in PATH; this is bad practise, and fortunately nothing seems to rely on it. * Only try to build broken packages if requested * Try harder to unmount leftover linprocfs mounts in the chroot, by copying in the 5.x mount binary and supporting libraries from the host system. The 5.x mount is able to unmount by FSID in situations where the 4.x umount becomes confused. * Don't clean up when we are signalled, that is done by the build master from outside. * Suppress some code relating to jail builds, which are not yet ready for use. * Don't push results of the build back to the master; the master now pulls them from the client when the build completes. Clients no longer need ssh access into the master; this is good for security as well as significantly reducing the load on the master since it is not thrashed by dozens of sshd processes.
* Hack around the sparc64 64-bit time_t test in 'installworld' sincekris2004-07-141-0/+1
| | | | | we're installing (into a temporary staging area) on a non-sparc64 machine that does not have 64-bit time_t.
* * Break out the 'parallel' target from /usr/ports/Makefile. Onekris2004-07-141-6/+40
| | | | | | | | | | | | | | | | | | | | | | advantage is that here we know the value of PKGSUFFIX (.tgz/.tbz) for the build via buildenv. * Add a list of 'quickports', which are ports with long dependency chains that we should kick off straight away to try and avoid bottlenecks later on when most of the cluster idles waiting for one or two ports to build. Ideally we'd build dependencies of these ports exclusively first and only build other ports when we run out (i.e. a build slot becomes free), but I couldn't work out how to do this. As a compromise, we now do 'make -k -j<#> quickports all' which doesn't give quite as high a priority to the quickports (i.e. we also build other ports from the beginning while there are quickport dependencies still to build), but is better than nothing. * Pass in the FETCH/EXTRACT/PATCH/BUILD/RUN_DEPENDS separately via env variables when dispatching a job. This allows us to add and remove the dependencies at the corresponding build stage to catch ports with dependencies listed too early/late.
* Set INDEX_JOBS to control index build concurrencykris2004-07-141-0/+3
|
* Drop the ignorelist concurrency back to 1 because I'm occasionally seeingkris2004-07-141-1/+1
| | | | | overlapping I/O. Something similar to how parallel index builds are now done will need to be done here too.
* * If we time out connecting to a machine, give it a weight of 999 to makekris2004-07-141-2/+2
| | | | | | | sure we don't try and schedule jobs on it even if all other machines are busy * Remove sleep in outer loop, this isn't needed or worthwhile now that there are so many machines being monitored
* Chase bento to pointyhatkris2004-07-141-1/+1
|
* * Be even more explicit about partial ports trees being unsupportedkris2004-07-141-13/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | for INDEX builds [1] * Remove the parallel target from Makefile; this is heavily tied to the package build cluster and can be better done in the makeparallel script (commit to follow) [2] * Extend the format of INDEX to separately list the EXTRACT/PATCH/FETCH_DEPENDS instead of lumping them all in together with BUILD_DEPENDS. The three new fields are appended to the end of the record in that order. [2] * Change BROKEN to IGNORE in BROKEN_WITH_MYSQL failure code [3] * Support non-default PREFIX for perl 5.00503 [5] * Use pkg_info -I instead of ls when searching for conflicts [6] * Allow local customization of the port subdirectories by including ${.CURDIR}/Makefile.local in bsd.subdir.mk if it exists [7] * Fix 'make search' when ${PORTSDIR} is a symlink to a directory name containing extended regexp metacharacters [8] Submitted by: linimon [1] [3], kris [2], lth [4], sem [5], eik [5] [6], Roman Neuhauser <neuhauser@chello.cz> [7] PR: 68299 [1], 67705 [3], 67264 [4], 59696 [5], 66568 [6], 68072 [7]
* Modernize these two scripts a little bit to make it easier to dealmurray2004-06-092-6/+16
| | | | | | with 5.X or 4.X package sets (i.e., INDEX vs INDEX-5 and tgz vs tbz) OKed by: marcus
* Hack in master/slave supporteik2004-06-051-27/+69
|
* Add dopackages.wrapper, a wrapper script around dopackages that sets upmarcus2004-06-031-0/+53
| | | | | | | | | | | | build locking, log files, and cleans things up if a build fails. This script is the primary starting point for a package build. Symlinks should be created in the form of dopackages.${branch} -> dopackages.wrapper where ${branch} is currently one of 4, 4-exp, or 5. This script takes the place of the unofficial (i.e. uncommitted) dopackages.steveX scripts. Ok'd by: kris Tested by: 4.10-RELEASE package build
* parse $FreeBSD$ with yyyy-mm-dd format dateseik2004-05-311-1/+1
|
* protect $FreeBSD: ... $ from being expandedeik2004-05-301-1/+1
| | | | Noticed by: mat
* don't terminate the log entry with the first hypheneik2004-05-301-1/+1
|
* - make somewhat more robust against a corrupted ports treeeik2004-05-301-30/+80
| | | | | | | | | | - CC committers and maintainer [1] - include affected ports in the subject line [2] - do a CVS log of the version checked out [3] Suggsted by: Ade Lovett <ade@FreeBSD.org> [1] Bjoern A. Zeeb <bzeeb-lists@lists.zabbadoz.net> [2] Pav Lucistnik <pav@FreeBSD.org> [3]
* - Remove 'pkg-plist' from required fileskrion2004-05-271-2/+1
| | | | Approved by: sobomax
* - fix a bug handling the maintainer field when ALLPORTS is usedeik2004-05-221-35/+56
| | | | | | - add an option to watch for maintainer changes - change watch regex to match PKGORIGIN, not PKGNAME
* Turn chkversion into an all-dancing, all-singing notification tool.eik2004-05-191-35/+138
| | | | | | | | You can even get notified of version changes in your favourite perl modules by setting WATCH_REGEX='p5-.*' Plus, it has a nice configurable nagging option.
* small optimazation: don't compare versions that are equaleik2004-05-181-1/+1
|
* check only ports connected to the build.eik2004-05-181-15/+32
|
* - integrated chkorigineik2004-05-171-51/+102
| | | | | | - eliminated shell usage [1] Submitted by: mat [1]
* FWIIW, spawn fewer processes.eik2004-05-161-3/+12
|
* support for symlinked portsdir (but no other symlinks in the ports tree)eik2004-05-152-14/+19
| | | | chkversion: support for CVS blame with env CVSBLAME=yes
* add support for ignoring local ports / categories:eik2004-05-151-1/+4
| | | | env 'EXCLUDE=local devel/local' /usr/ports/Tools/scripts/chkmodules.pl
* added chkversion, a script that ensures that PORTVERSIONs don't go backwardseik2004-05-141-0/+155
|
* add a perl script to check CVSROOT-ports/modules,eik2004-05-141-0/+127
| | | | based on a shell script by Clement Laforet.
* - don't flag empty lineseik2004-05-081-6/+30
| | | | | | - callable without arguments: /usr/ports/Tools/scripts/MOVEDlint.awk - do CVS annotate when called with env CVS=yes
* awk script to check MOVED for consistencyeik2004-05-081-0/+87
| | | | | Usage: [env PORTSDIR=/usr/ports] awk -f MOVEDlint.awk /usr/ports/MOVED
* While generating commit message, don't lowercase the first letter if the secondmat2004-04-051-2/+2
| | | | | | | is upper case (ie : don't end up with pEAR) Approved by: will Asked by: thierry
* Set INDEX_PRISTINE and INDEX_QUIET to catch up to new index buildkris2004-04-021-3/+5
| | | | variables.
* added an -q option to enable usage in crontab.eik2004-03-291-2/+27
| | | | Suggested by: erwin
* Fix a bug arising when you added a port which should have been at the end ofmat2004-03-211-3/+10
| | | | | | <category>/Makefile, and ended up at the begining Approved by: will
* Add chkorigin.sh, a script that checks the ports tree for ports with a wrongeik2004-03-192-0/+89
| | | | | | | | PKGORIGIN. Useful especially after repocopies. env PORTSDIR=/usr/ports sh /usr/ports/Tools/scripts/chkorigin.sh Approved by: marcus (portmgr)
* Fill /mnt/ before each port and clear it afterward, to avoid runningtrevor2004-03-161-1/+2
| | | | out of inodes.
* This script had served its purpose.knu2004-03-141-146/+0
|
* Add note about FETCH_BEFORE_ARGS=-s.trevor2004-03-131-0/+4
|
* script for finding "size mismatch" and "size unknown" errorstrevor2004-03-131-0/+58
|
* Run a command on all client machines of a certain architecture, or allkris2004-03-081-0/+50
| | | | client machines.
* Walk through a package set and prune files that are not listed in INDEXkris2004-03-081-0/+66
| | | | | (e.g. packages from a previous build run, packages that incorrectly modify their PORTNAME at install-time, etc.) as well as dangling symlinks.
* Verify the archive integrity of a package set, and move aside any thatkris2004-03-081-0/+45
| | | | | are found to be bad. Not currently in use, since we do not experience package corruption and this takes quite a while to run.
* To be run on the client, this script looks for chroot directories that have ↵kris2004-03-081-0/+48
| | | | | | | | not been used in 20 minutes, as well as directories listed as 'in use' that have not been touched in 24 hours (corresponding to port builds that have timed out or shut down uncleanly) and prunes them to reclaim space. This is intended to be run as a cron job.
* Break out a script from dopackages for building the restricted.sh filekris2004-03-081-0/+34
| | | | so it can be easily done manually
* Replace gnome with gnome2 in quickports. I'm still not sure this actuallykris2004-03-081-1/+1
| | | | works as intended.
* Remove the integrity testing code now that the corruption bug has beenkris2004-03-081-4/+0
| | | | identified and those machines removed (restricted to Miatas running 5.2)
* * Fix 4.x builds with 5.x host systems by copying some additional neededkris2004-03-081-0/+6
| | | | files into the chroot to support host ps and killall binaries.
* Expand out some shell globbing syntaxkris2004-03-081-1/+1
|
* Prune out packages that will not be built because a dependency is markedkris2004-03-081-1/+1
| | | | IGNORE/FORBIDDEN/etc (using duds.full).
* Create a duds.full file that lists not only files that are marked IGNOREkris2004-03-081-0/+4
| | | | but everything that depends on them.
* Increase timeout on slow build machines (alpha, sparc64) to 16 hours.kris2004-03-081-2/+8
| | | | This should probably be made a per-machine variable.
* * Set the PACKAGE_BUILDING variable in the build environment so ports cankris2004-03-081-1/+4
| | | | | detect they are being built on bento * Update the list of "extra files" to filter to reduce some false positives
* Set up the DISTDIR and PACKAGES variables for use by client scriptskris2004-03-081-0/+3
|
* Add sunshar, a "secure unshar" for ports committers, which:knu2004-02-283-0/+368
| | | | | | | | | | | - Does not execute unknown commands nor call sh(1) at all. - Does not overwrite existing files by default. - Does not extract files into upper directories. - Does have a dry run (-n) flag to see what would have been extracted. - Does have a strip (-p N) flag to strip any number of levels from pathnames. It (so far) only supports shell archives made with BSD shar.
* Update "plist" to understand the rest of the mtree(5) file format,green2004-02-281-14/+36
| | | | | thus fixing the extra-@dirrm-problem once the base mtree files document the extra symlinks that are part of the local/etc. trees.
* Fix up the ordering and remove some duplicates so the resulting summary ismarcus2004-02-041-22/+26
| | | | | | | | | | | | | | | | | more accurate: * The tests for "Cannot stat", "can't cd", and "tar command failed" are moved to the bottom. This will sacrifice a little bit of performance for a gain in accuracy * The tests for "configuration errors" needed to be combined and tweaked to better identify errors that are architecture-specific (usuallly the "all pointers are ints" assumption) * The detection of chown was buggy * There were some duplicate test cases * As an experiment, another test for dependency problems is added PR: 61914 Submitted by: linimon
* - Fix synopsisale2004-01-231-2/+2
| | | | | | - Add 'ports/' PR category to commit log Approved by: will
* Comment out the munmap reason since it is quite unlikely this will bemarcus2004-01-221-4/+4
| | | | | | | | | encountered again, nad having it in there slows performance. However, I'm leaving it in the file in case a bad bindist slips back in, and it becomes convenient to uncomment this for purposes of seeing what build failures are cluster-related. Requested by: kris
* Add domakedescribe and doportlint, two scripts that iterate over the portsmarcus2004-01-202-0/+39
| | | | | | | | tree, and run "make describe" and portlint respectively. They can be useful as automated linting tools. PR: 59226 Submitted by: linimon
* * Refactor processlogs so that new error types can be added easily to a newmarcus2004-01-203-312/+441
| | | | | | | | | | | | | | | processonelog script, that contains all the per-errorlog logic [1] * Add new error checks, one for detecting CPU problems, and the other for checking for broken libgnugetopt support [2] * Add a new "munmap" error type to check for broken bindists * Add imake and pthread detection to processonelog [3] * Add errortype column to the bento "New Build Failures" report [4] PR: 50258 56859 [1] 54406 [2] 57067 [3] 59272 [4] Submitted by: linimon [1] [2] [3] [4]
* Add security-check.awk, a more efficient implementation of the portsmarcus2004-01-201-0/+100
| | | | | | | | system's security checking algorithm. This will be used in the upcoming changes to bsd.*.mk. PR: 55331 Submitted by: Eugene M. Kim <ab@astralblue.com>
* * Add support for linprocfs (needed for JDK ports) [1]marcus2004-01-201-2/+23
| | | | | | | * Be smarter about adding directories to ldconfig * Add code to test package integrity [1] Submitted by: kris [1]
* Use editors/openoffice-1.1 instead of editors/openoffice since the latter ismarcus2004-01-161-1/+1
| | | | now gone.
* When suggesting to add new port's submitter to Contributors' List,demon2004-01-081-1/+2
| | | | | | | | also print the exact location of the file to checkout. This is for those who rarely adds new ports and do not remember the location of Additional FreeBSD Contributors article. Approved by: will
* Refactor how we format the autofill message. With the slightly questionablepetef2004-01-061-34/+15
| | | | | | | | Perl before, sometimes we would word wrap too soon. Now we should never generate a line more than 72 characters long (but as close to 72 characters long as we can without breaking up a word). Approved by: will (maintainer)
* There is nothing "future" about 'addport'.obrien2004-01-031-1/+1
|
* Use 'make master-sites-all' instead of 'make -V MASTER_SITES' to obtain the listdemon2003-12-171-1/+1
| | | | | | | | of master sites. Before that %SUBDIR% was not expanded and the resulting URL was malformed. L-o-o-o-ng maintainer timeout waiting for review of a slightly different solution.
* fetch(1)'s -b and -t options were deprecated years ago, so stop using them.des2003-11-121-1/+1
|
* Condense the table a bitkris2003-10-131-3/+3
|
* The newfailure file is no longer updated by buildskris2003-10-131-3/+0
|
* Script for migrating from old-style to new-style failure log file.kris2003-10-131-0/+74
|
* * newfailure is no longer updated, no need to prune itkris2003-10-131-42/+6
| | | | * Update failure file format with both original and latest failure date
* * Generate newfailure file (sorted list by original failure date) from failurekris2003-10-131-10/+13
| | | | | * Enhance HTML output to include both original and latest failure date, and number of times the build has failed.
* * Try to be more careful when writing the failure file in case the diskkris2003-10-131-13/+16
| | | | | | | | is full * The newfailure file is no longer maintained, but generated as-needed by the processfail script * Update format of failure file (records both initial failure date and most recent failure date)
* Add a script that prunes the contents of the failure and newfailure fileskris2003-10-131-0/+114
| | | | | | | | of stale entries (removed ports, ports listed in duds that were mistakenly built, malformed entries) and refreshes the version of entries to the latest in INDEX. This must be run under lockf (see the comments in the script) to avoid racing with portbuild which also tries to write to these files.
* Fix -l option comment removing obsolete pkg-comment reference.bland2003-09-291-2/+2
| | | | Approved by: will
* Teach this script about different architectures.kris2003-09-132-61/+54
|
* Add amd64 to the list of architectures to poll.kris2003-09-131-1/+1
|
* Add a couple of amd64 failure messages.kris2003-09-131-0/+6
|
* Remove some ldconfig -R's that were apparently confusing things, coalescekris2003-09-131-6/+3
| | | | | some ldconfig -m's into one command, and run buildscript niced in the chroot build case, not just the (currently unused) jailed case.
* Set PACKAGE_BUILDING before making the duds file to prevent listing ofkris2003-09-131-0/+1
| | | | ports that are buildable conditionally on this variable.
* Export MASTER_SITE_OVERRIDE with DIST_SUBDIR correctly escapedkris2003-09-131-2/+1
|
* Catch up to makefile changes: build in etc/ before trying to installkris2003-09-131-0/+2
| | | | | | from it. Submitted by: ru
* When using addport on freefall, there's no need to ssh freefall once more tomat2003-09-021-1/+5
| | | | | | use modulesupdate. Approved by: will
* Add a -fetch-original switch to dopackages that fetches the distfiles fromkris2003-05-183-1/+16
| | | | | | the listed MASTER_SITES instead of ftp-master. Pass the FETCH_ORIGINAL environment variable to pdispatch and as a command-line switch to portbuild.
* Only transfer the build distfiles to bento if requested.kris2003-05-171-7/+14
| | | | | Switch back to chroot builds until I can debug the remaining problems with jail builds.
* Pass down the request to keep distfiles if appropriate.kris2003-05-171-0/+3
|
* Add a -distfiles flag to specify that we should retain distfiles for laterkris2003-05-171-4/+12
| | | | | | | | processing (e.g. transfer to ftp site). It was taking up too much disk space to collect distfiles for 14 architecture/version combinations (including the backup set from the previous run). Put the cvsdone files in the arch/branch subdirectory, not arch/.
* Switch to polling the clients with netcat to collect load stats, insteadkris2003-05-171-13/+10
| | | | | | of having the clients scp their loads to bento every 10 seconds. Fix some indentation and add some sleeps to make sure the startup script doesn't run too early in the boot process.
* Previous delta was:sobomax2003-04-181-1/+1
| | | | Submitted by: ru
* Clean up empty dirs.sobomax2003-04-181-2/+2
|
* Speed-up a bit when user keeps his working directories in the ports treesobomax2003-04-081-2/+5
| | | | | | (doesn't use WRKDIRPREFIX). Submitted by: ru
* Update sed pattern used to process logskris2003-04-061-1/+1
|
* Zap some old/non-useful codekris2003-04-061-108/+106
| | | | | | | | | Add some new command-line options: * -continue : restart the build without rebuilding ports that have already failed * -nofinish : don't run the post-processing steps when the build finishes * -finish : run the post-processing steps only Reorder some code sections appropriately
* This tool parses the output of kdump to generate a list of addededwin2003-04-041-0/+199
| | | | | | | | | | | | | and removed files. This can be used as the basis of a pkg-plist, or even just for curiosity about what files something is touching. Fairly raw at the moment, and doubtless inefficient, but it should make a useful tool for port creators. PR: ports/47424 Submitter: Daniel O'Connor <doconnor@gsoft.com.au>
* Sort the non-dirrm entries, and reverse sort the dirrm entries.marcus2003-03-281-1/+5
| | | | Reviewed by: green
generated by cgit (git 2.34.1) at 2024-12-29 13:10:12 +0800