ports-mgmt/portscout: Add GitHub and PyPI site handlers & MORE!

Over the past several months portscout.freebsd.org appears to have been
more frequently NOT finding updates, particularly for ports that use
CHEESESHOP (PyPI) as their MASTER_SITES.

Portscout has also never worked for ports using GitHub for distribution
files due to the following:

  a) Portscout, prior to 'guessing', requests a randomly named file
     from the Site and expects a 4xx (404) in response. If it doesn't
     receive a 4xx response, it increments a 'lie counter' and does not
     check the site again in the next run.
  b) The GitHUB handlers (SUBDIR/MASTER_SITES) in bsd.sites.mk
     construct a URL that ends in a a dummy query paramater (for the
     filename), so that fetch saves the correct filename to DISTDIR.
     This means for any DISTFILE name provided, a 200 OK response is
     returned

These two factors unfortunately leave us in a position where there is no
good way to workaround this in the ports framework, including overriding
DISTFILES, DISTNAME, FETCH_ARGS, or the SUBDIR URL itself for various
reasons (not matching distinfo, file conflicts in DISTDIR, etc)

Fortunately, the portroach project (OpenBSD's fork of portscout)
contains a site handler for GitHub and PyPI (among others) already [1].

These site handlers use API endpoints at GitHub and PyPI that respond
JSON respectively, providing a faster and more accurate way to determine
the latest version of a package, without having to go through the
'guessing' process.

This commit:

- Adds GitHub and PyPI site handlers, and modifies or extends them to
  accept/match our MASTER_SITES URL's.
- Adds authenticated API request support and two settings for the
  GitHub site handler
- Add p5-JSON to RUN_DEPENDS (needed by new site handlers)
- Add HTTPS option for supporting https:// MASTER_SITES. Currently
  portscout does not check (fails) https:// MASTER_SITES [2]
- Take MAINTAINER'ship
- Adds badly needed logging/debugging messages to key parts of the
  process retaining the conditional logic that ties the verbosity to
  "quiet" or "debug" portscout.conf settings.
- Renables the SQLITE3 option (previously commented out) and renames it
  to SQLITE (the standard, as per bsd.options.desk.mk)
- Creates a DATABASE option group allowing either/or SQLITE or
  POSTGRESQL to be selected
- Switches option conditionals where possible to options helpers
- Backport a fix for maintainer matching/mapping [3]
- Adds LICENSE (BSD2CLAUSE)
- Updates and sorts pkg-plist

This change was tested again ports maintained by me, and resulted in
'new versions' being found and reported for 42 of my ports (of 123).

[1] https://github.com/jasperla/portroach/tree/master/Portroach/SiteHandler
[2] Reported by: truckman
[3] https://github.com/jasperla/portroach/commit/2f6ee134ddc178f74688b37b986b66ecb0481782

PR:			203996
Approved by:		maintainer (timeout, 15 days)
Differential Revision:	https://reviews.freebsd.org/D3996

10 files changed, 568 insertions, 34 deletions

diff --git a/ports-mgmt/portscout/Makefile b/ports-mgmt/portscout/Makefile
index 8d62802e1eb9..96d9444c09db 100644
--- a/ports-mgmt/portscout/Makefile
+++ b/ports-mgmt/portscout/Makefile
@@ -3,44 +3,52 @@
 
 PORTNAME=	portscout
 PORTVERSION=	0.8.1
-PORTREVISION=	3
+PORTREVISION=	4
 CATEGORIES=	ports-mgmt
-MASTER_SITES=	http://mirror.inerd.com/FreeBSD/distfiles/${PORTNAME}/	\
-		http://www.atarininja.org/~wxs/distfiles/		\
+MASTER_SITES=	http://mirror.inerd.com/FreeBSD/distfiles/${PORTNAME}/ \
+		http://www.atarininja.org/~wxs/distfiles/ \
 		http://www.inerd.com/software/${PORTNAME}/
 
-MAINTAINER=	shaun@FreeBSD.org
+MAINTAINER=	koobs@FreeBSD.org
 COMMENT=	Tool to scan for new versions of FreeBSD ports
 
-OPTIONS_DEFINE=	SQLITE3 DOCS
-SQLITE3_DESC=	Use SQLite backend instead of PostgreSQL
+LICENSE=	BSD2CLAUSE
 
+RUN_DEPENDS=	p5-DBI>=0:${PORTSDIR}/databases/p5-DBI \
+		p5-Proc-Queue>=0:${PORTSDIR}/devel/p5-Proc-Queue \
+		p5-Net>=0:${PORTSDIR}/net/p5-Net \
+		p5-URI>=0:${PORTSDIR}/net/p5-URI \
+		p5-XML-XPath>=0:${PORTSDIR}/textproc/p5-XML-XPath \
+		p5-MIME-Lite>=0:${PORTSDIR}/mail/p5-MIME-Lite \
+		p5-libwww>=0:${PORTSDIR}/www/p5-libwww \
+		p5-JSON>0:${PORTSDIR}/converters/p5-JSON
+
+OPTIONS_DEFINE=		DOCS HTTPS
+OPTIONS_DEFAULT=	HTTPS PGSQL
+
+OPTIONS_MULTI=		DATABASE
+OPTIONS_MULTI_DATABASE=	SQLITE PGSQL
+
+DATABASE_DESC=	Database Backends
 USES=		perl5 shebangfix
+
 NO_BUILD=	yes
 SHEBANG_FILES=	${WRKSRC}/portscout.pl
 
 PORTDOCS=	UPDATING portscout-portconfig.txt xml-datasrc-example.xml
 
-RUN_DEPENDS=	p5-DBI>=0:${PORTSDIR}/databases/p5-DBI	\
-		p5-Proc-Queue>=0:${PORTSDIR}/devel/p5-Proc-Queue	\
-		p5-Net>=0:${PORTSDIR}/net/p5-Net			\
-		p5-URI>=0:${PORTSDIR}/net/p5-URI			\
-		p5-XML-XPath>=0:${PORTSDIR}/textproc/p5-XML-XPath	\
-		p5-MIME-Lite>=0:${PORTSDIR}/mail/p5-MIME-Lite		\
-		p5-libwww>=0:${PORTSDIR}/www/p5-libwww
+HTTPS_RUN_DEPENDS=	p5-LWP-Protocol-https>=0:${PORTSDIR}/www/p5-LWP-Protocol-https
 
-.include <bsd.port.options.mk>
+SQLITE_USE=		SQLITE=3
+SQLITE_RUN_DEPENDS=	p5-DBD-SQLite>=0:${PORTSDIR}/databases/p5-DBD-SQLite
 
-.if ${PORT_OPTIONS:MSQLITE3}
-#USE_SQLITE=	3
-#RUN_DEPENDS+=	p5-DBD-SQLite>=0:${PORTSDIR}/databases/p5-DBD-SQLite
-.elif !defined(WITHOUT_PGSQL)
-#USES+=		pgsql
-#RUN_DEPENDS+=	p5-DBD-Pg>=0:${PORTSDIR}/databases/p5-DBD-Pg
-.endif
+PGSQL_USES=		pgsql
+PGSQL_RUN_DEPENDS=	p5-DBD-Pg>=0:${PORTSDIR}/databases/p5-DBD-Pg
+
+.include <bsd.port.options.mk>
 
 pre-everything::
-.if ${PORT_OPTIONS:MSQLITE3}
+.if ${PORT_OPTIONS:MSQLITE}
 	@${ECHO_MSG} "+-------------------------------------------------------------+"
 	@${ECHO_MSG} "| Warning! although SQLite is supported, portscout will only  |"
 	@${ECHO_MSG} "| operate in non-forking mode with this database backend. It  |"
@@ -48,8 +56,12 @@ pre-everything::
 	@${ECHO_MSG} "+-------------------------------------------------------------+"
 .endif
 
+post-extract:
+	@${CP} ${FILESDIR}/files-Portscout-SiteHandler-GitHub.pm ${WRKSRC}/Portscout/SiteHandler/GitHub.pm
+	@${CP} ${FILESDIR}/files-Portscout-SiteHandler-PyPI.pm ${WRKSRC}/Portscout/SiteHandler/PyPI.pm
+
 post-patch:
-.if ${PORT_OPTIONS:MSQLITE3}
+.if ${PORT_OPTIONS:MSQLITE}
 	@${REINPLACE_CMD} 's/^\([^#]*DBI:Pg.*\)$$/#\1/g' ${WRKSRC}/portscout.conf
 	@${REINPLACE_CMD} 's/^#\(.*DBI:SQLite.*\)$$/\1/g' ${WRKSRC}/portscout.conf
 .endif
@@ -96,7 +108,7 @@ do-install:
 test:
 	${PERL} ${WRKSRC}/t/00-use.t
 	${PERL} ${WRKSRC}/t/01-vercompare.t
-.if ${PORT_OPTIONS:MSQLITE3}
+.if ${PORT_OPTIONS:MSQLITE}
 	${PERL} ${WRKSRC}/t/10-sqlite.t
 #.else
 #	${PERL} ${WRKSRC}/10-postgresql.t
diff --git a/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm b/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm
new file mode 100644
index 000000000000..7de8a9ecfbe7
--- /dev/null
+++ b/ports-mgmt/portscout/files/files-Portscout-SiteHandler-GitHub.pm
@@ -0,0 +1,175 @@
+#------------------------------------------------------------------------------
+# Copyright (C) 2014, Jasper Lievisse Adriaanse <jasper@openbsd.org>
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+#------------------------------------------------------------------------------
+
+package Portscout::SiteHandler::GitHub;
+
+use JSON qw(decode_json);
+use LWP::UserAgent;
+
+use Portscout::Const;
+use Portscout::Config;
+
+use strict;
+
+require 5.006;
+
+
+#------------------------------------------------------------------------------
+# Globals
+#------------------------------------------------------------------------------
+
+push @Portscout::SiteHandler::sitehandlers, __PACKAGE__;
+
+our %settings;
+
+
+#------------------------------------------------------------------------------
+# Func: new()
+# Desc: Constructor.
+#
+# Args: n/a
+#
+# Retn: $self
+#------------------------------------------------------------------------------
+
+sub new
+{
+	my $self      = {};
+	my $class     = shift;
+
+	$self->{name} = 'GitHub';
+
+	bless ($self, $class);
+	return $self;
+}
+
+
+#------------------------------------------------------------------------------
+# Func: CanHandle()
+# Desc: Ask if this handler (package) can handle the given site.
+#
+# Args: $url - URL of site.
+#
+# Retn: $res - true/false.
+#------------------------------------------------------------------------------
+
+sub CanHandle
+{
+	my $self = shift;
+
+	my ($url) = @_;
+
+	return ($url =~ /^https?:\/\/([^\/.]+\.)?github\.com\/(.*?)\/tar.gz/);
+}
+
+
+#------------------------------------------------------------------------------
+# Func: GetFiles()
+# Desc: Extract a list of files from the given URL. In the case of GitHub,
+#       we are actually pulling the files from the project's Atom feed and
+#       extract the release url, containing the tag it was based on.
+#
+# Args: $url     - URL we would normally fetch from.
+#       \%port   - Port hash fetched from database.
+#       \@files  - Array to put files into.
+#
+# Retn: $success - False if file list could not be constructed; else, true.
+#------------------------------------------------------------------------------
+
+sub GetFiles
+{
+	my $self = shift;
+
+	my ($url, $port, $files) = @_;
+	my $projname;
+
+	if ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) {
+	    $projname = $1;
+	} elsif ($url =~ /https:\/\/github.com\/downloads\/(.*)\//) {
+	    $projname = $1;
+	}
+
+	if ($projname) {
+		my ($query, $ua, $response, $items, $json);
+
+		# First check if there's a latest releases endpoint
+		$query = 'https://api.github.com/repos/' . $projname . '/releases/latest';
+
+		_debug("GET $query");
+		$ua = LWP::UserAgent->new;
+		$ua->agent(USER_AGENT);
+		$ua->timeout($settings{http_timeout});
+
+		$response = $ua->request(HTTP::Request->new(GET => $query));
+
+		if (!$response->is_success || $response->status_line !~ /^2/) {
+			_debug('GET failed: ' . $response->status_line);
+			# Project didn't do any releases, so let's try tags instead.
+			$query = 'https://api.github.com/repos/' . $projname . '/tags';
+			_debug("GET $query");
+			$ua = LWP::UserAgent->new;
+			$ua->agent(USER_AGENT);
+			$ua->timeout($settings{http_timeout});
+
+			$response = $ua->request(HTTP::Request->new(GET => $query));
+
+			if (!$response->is_success || $response->status_line !~ /^2/) {
+			    _debug('GET failed: ' . $response->status_line);
+			    return 0;
+			}
+
+			$json = decode_json($response->decoded_content);
+			foreach my $tag (@$json) {
+			    my $tag_url = $tag->{tarball_url};
+			    push(@$files, $tag_url);
+			}
+
+			_debug('Found ' . scalar @$files . ' files');
+			return 1;
+		}
+
+		$json = decode_json($response->decoded_content);
+		push(@$files, $json->{tarball_url});
+
+		_debug('Found ' . scalar @$files . ' files');
+	} else {
+		return 0;
+	}
+
+	return 1;
+}
+
+
+#------------------------------------------------------------------------------
+# Func: _debug()
+# Desc: Print a debug message.
+#
+# Args: $msg - Message.
+#
+# Retn: n/a
+#------------------------------------------------------------------------------
+
+sub _debug
+{
+	my ($msg) = @_;
+
+	$msg = '' if (!$msg);
+
+	print STDERR "(" . __PACKAGE__ . ") $msg\n" if ($settings{debug});
+}
+
+1;
diff --git a/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm b/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm
new file mode 100644
index 000000000000..c63b8ca6f68e
--- /dev/null
+++ b/ports-mgmt/portscout/files/files-Portscout-SiteHandler-PyPI.pm
@@ -0,0 +1,147 @@
+#------------------------------------------------------------------------------
+# Copyright (C) 2015, Jasper Lievisse Adriaanse <jasper@openbsd.org>
+#
+# Permission to use, copy, modify, and distribute this software for any
+# purpose with or without fee is hereby granted, provided that the above
+# copyright notice and this permission notice appear in all copies.
+#
+# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+#
+#------------------------------------------------------------------------------
+
+package Portscout::SiteHandler::PyPI;
+
+use JSON qw(decode_json);
+use LWP::UserAgent;
+
+use Portscout::Const;
+use Portscout::Config;
+
+use strict;
+
+require 5.006;
+
+
+#------------------------------------------------------------------------------
+# Globals
+#------------------------------------------------------------------------------
+
+push @Portscout::SiteHandler::sitehandlers, __PACKAGE__;
+
+our %settings;
+
+
+#------------------------------------------------------------------------------
+# Func: new()
+# Desc: Constructor.
+#
+# Args: n/a
+#
+# Retn: $self
+#------------------------------------------------------------------------------
+
+sub new
+{
+	my $self      = {};
+	my $class     = shift;
+
+	$self->{name} = 'PyPI';
+
+	bless ($self, $class);
+	return $self;
+}
+
+
+#------------------------------------------------------------------------------
+# Func: CanHandle()
+# Desc: Ask if this handler (package) can handle the given site.
+#
+# Args: $url - URL of site.
+#
+# Retn: $res - true/false.
+#------------------------------------------------------------------------------
+
+sub CanHandle
+{
+	my $self = shift;
+
+	my ($url) = @_;
+
+	return ($url =~ /https?:\/\/pypi\.python\.org\//);
+}
+
+
+#------------------------------------------------------------------------------
+# Func: GetFiles()
+# Desc: Extract a list of files from the given URL. Simply query the API.
+#
+# Args: $url     - URL we would normally fetch from.
+#       \%port   - Port hash fetched from database.
+#       \@files  - Array to put files into.
+#
+# Retn: $success - False if file list could not be constructed; else, true.
+#------------------------------------------------------------------------------
+
+sub GetFiles
+{
+	my $self = shift;
+
+	my ($url, $port, $files) = @_;
+
+	my ($pypi, $package, $resp, $query, $ua);
+	$pypi = 'https://pypi.python.org/pypi/';
+
+	# Strip all the digits at the end to keep the stem of the module.
+	if ($port->{distname} =~ /(.*?)-(\d+)/) {
+	    $package = $1;
+	}
+
+	$query = $pypi . $package . '/json';
+
+	_debug("GET $query");
+	$ua = LWP::UserAgent->new;
+	$ua->agent(USER_AGENT);
+	$resp = $ua->request(HTTP::Request->new(GET => $query));
+	if ($resp->is_success) {
+	    my ($json, $info, $version);
+
+    	    $json = decode_json($resp->decoded_content);
+	    $info = $json->{info};
+	    $version = $info->{version};
+	    next unless $version;
+
+	    push(@$files, $json->{releases}{$version}[0]{filename});
+	} else {
+	    _debug("GET failed: " . $resp->code);
+	    return 0;
+	}
+
+	return 1;
+}
+
+
+#------------------------------------------------------------------------------
+# Func: _debug()
+# Desc: Print a debug message.
+#
+# Args: $msg - Message.
+#
+# Retn: n/a
+#------------------------------------------------------------------------------
+
+sub _debug
+{
+	my ($msg) = @_;
+
+	$msg = '' if (!$msg);
+
+	print STDERR "(" . __PACKAGE__ . ") $msg\n" if ($settings{debug});
+}
+
+1;
diff --git a/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm b/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm
new file mode 100644
index 000000000000..867a0ab0fde9
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-Portscout_DataSrc_Ports.pm
@@ -0,0 +1,81 @@
+--- Portscout/DataSrc/Ports.pm.orig	2011-04-09 17:19:03 UTC
++++ Portscout/DataSrc/Ports.pm
+@@ -201,7 +201,7 @@ sub BuildDB
+ 
+ 	my $lastbuild = getstat('buildtime', TYPE_INT);
+ 
+-	print "Looking for updated ports...\n\n"
++	print "Incremental build: Looking for updated ports...\n\n"
+ 		if ($incremental);
+ 
+ 	$got_ports = 0;
+@@ -243,12 +243,15 @@ sub BuildDB
+ 			my (@fields, $maintainer, $port);
+ 
+ 			@fields = split /\|/;
+-			$maintainer = $fields[5];
++			$maintainer = lc($fields[5]);
+ 			$port = $fields[1];
+ 			$port =~ s/^(?:.*\/)?([^\/]+)\/([^\/]+)$/$1\/$2/;
+ 
+-			$portsmaintok{$port} = $maintainer
+-				if ($maintainers{$maintainer});
++			if ($maintainers{$maintainer}) {
++				$portsmaintok{$port} = $maintainer;
++				print "Maintainer match: $maintainer $port \n"
++					unless ($settings{quiet});
++			}
+ 		}
+ 
+ 		close $if;
+@@ -264,7 +267,7 @@ sub BuildDB
+ 
+ 		opendir my $catdir, $settings{ports_dir}."/$cat";
+ 
+-		print "Scanning $cat...\n"
++		print "Scanning $cat ...\n"
+ 			unless ($settings{quiet});
+ 
+ 		while (my $name = readdir $catdir) {
+@@ -276,9 +279,9 @@ sub BuildDB
+ 			# port directory's mtime; skip if not updated.
+ 			if ($incremental) {
+ 				my ($updated);
+-
+ 				opendir my $portdir, $settings{ports_dir}."/$cat/$name";
+-
++				print "Scanning $cat/$name ... "
++					unless ($settings{quiet});
+ 				while (my $subfile = readdir $portdir) {
+ 					my ($subfile_path, $fi);
+ 
+@@ -289,12 +292,18 @@ sub BuildDB
+ 						or die "Couldn't stat $subfile_path: $!";
+ 
+ 					if ($fi->mtime > $lastbuild) {
++						print "$subfile (mtime: $fi->mtime) modified updated since last build: $lastbuild \n"
++							if ($settings{debug});
+ 						$updated = 1;
+ 						last;
+ 					}
+ 				}
+ 
+-				next if (!$updated);
++				if (!$updated) {
++					print "Not modified since last build: $lastbuild \n"
++						if ($settings{debug});
++					next;
++				}
+ 			}
+ 
+ 			# Check this port is wanted by user
+@@ -307,7 +316,8 @@ sub BuildDB
+ 					&& $settings{indexfile_enable}) {
+ 				next if (!$portsmaintok{"$cat/$name"});
+ 			}
+-
++			print "Matched: $cat/$name\n"
++				unless ($settings{quiet});
+ 			push @ports, "$cat/$name";
+ 		}
+ 	}
diff --git a/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm
new file mode 100644
index 000000000000..8eb32476632c
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler.pm
@@ -0,0 +1,11 @@
+--- Portscout/SiteHandler.pm.orig	2010-04-29 01:07:51 UTC
++++ Portscout/SiteHandler.pm
+@@ -31,6 +31,8 @@ package Portscout::SiteHandler;
+ use XML::XPath;
+ use XML::XPath::XMLParser;
+ 
++use Portscout::SiteHandler::GitHub;
++use Portscout::SiteHandler::PyPI;
+ use Portscout::SiteHandler::SourceForge;
+ 
+ use strict;
diff --git a/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm
new file mode 100644
index 000000000000..7d7f9ea87504
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_GitHub.pm
@@ -0,0 +1,26 @@
+--- Portscout/SiteHandler/GitHub.pm.orig	2015-10-25 05:00:48 UTC
++++ Portscout/SiteHandler/GitHub.pm
+@@ -97,7 +97,9 @@ sub GetFiles
+ 	my ($url, $port, $files) = @_;
+ 	my $projname;
+ 
+-	if ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) {
++	if ($url =~ /https?:\/\/codeload\.github\.com\/(.*?)\/tar.gz\//) {
++	    $projname = $1;
++	} elsif ($url =~ /https:\/\/github\.com\/(.*?)\/archive\//) {
+ 	    $projname = $1;
+ 	} elsif ($url =~ /https:\/\/github.com\/downloads\/(.*)\//) {
+ 	    $projname = $1;
+@@ -108,7 +110,11 @@ sub GetFiles
+ 
+ 		# First check if there's a latest releases endpoint
+ 		$query = 'https://api.github.com/repos/' . $projname . '/releases/latest';
+-
++		# Add GitHub Client ID & Secret if they are set in settings
++		# https://developer.github.com/v3/#authentication
++		if ($settings{github_client_id} && $settings{github_client_id}) {
++			$query = $query . "?client_id=$settings{github_client_id}&client_secret=$settings{github_client_secret}";
++		}
+ 		_debug("GET $query");
+ 		$ua = LWP::UserAgent->new;
+ 		$ua->agent(USER_AGENT);
diff --git a/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm
new file mode 100644
index 000000000000..8070381f231a
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-Portscout_SiteHandler_PyPI.pm
@@ -0,0 +1,11 @@
+--- Portscout/SiteHandler/PyPI.pm.orig	2015-10-25 05:00:48 UTC
++++ Portscout/SiteHandler/PyPI.pm
+@@ -115,7 +115,7 @@ sub GetFiles
+ 	    $info = $json->{info};
+ 	    $version = $info->{version};
+ 	    next unless $version;
+-
++	    _debug("GET success: " . $resp->code . " Filename: " . $json->{releases}{$version}[0]{filename});
+ 	    push(@$files, $json->{releases}{$version}[0]{filename});
+ 	} else {
+ 	    _debug("GET failed: " . $resp->code);
diff --git a/ports-mgmt/portscout/files/patch-portscout.conf b/ports-mgmt/portscout/files/patch-portscout.conf
new file mode 100644
index 000000000000..7b6243d77eb0
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-portscout.conf
@@ -0,0 +1,14 @@
+--- portscout.conf.orig	2015-10-25 05:00:59 UTC
++++ portscout.conf
+@@ -172,5 +172,11 @@ db port          =           				# Port
+ #db connstr       = DBI:Pg:dbname=%(db_name);host=%(db_host);port=%(db_port)
+ db connstr       = DBI:SQLite:dbname=/var/db/portscout.db
+ 
++# GitHub site handler settings
++# GitHub rate limits requests to its API to a very low number for unauthenticated
++# requests, and 5000 per hour for authenticated requests.
++
++# github_client_id	=				# GitHub Client ID
++# github_client_secret	=				# GitHub Client Secret
+ 
+ # ex: ts=4 sw=4
diff --git a/ports-mgmt/portscout/files/patch-portscout.pl b/ports-mgmt/portscout/files/patch-portscout.pl
new file mode 100644
index 000000000000..a3295f6b2e50
--- /dev/null
+++ b/ports-mgmt/portscout/files/patch-portscout.pl
@@ -0,0 +1,55 @@
+--- portscout.pl.orig	2015-10-25 05:00:48 UTC
++++ portscout.pl
+@@ -463,7 +463,7 @@ sub VersionCheck
+ 
+ 		$i++;
+ 
+-		info($k, 'Checking site: ' . strchop($site, 60));
++		info($k, 'Checking site: ' . strchop($site, 200));
+ 
+ 		# Look to see if the URL contains the distfile version.
+ 		# This will affect our checks and guesses later on.
+@@ -493,15 +493,23 @@ sub VersionCheck
+ 		}
+ 
+ 		# Check for special handler for this site first
++		print "Does site handler exist ... "
++			unless ($settings{quiet});
+ 		if (my $sh = Portscout::SiteHandler->FindHandler($site))
+ 		{
+-			info($k, $site, 'Using dedicated site handler for site.');
++			print "Yes \n"
++				unless ($settings{quiet});
+ 
+ 			if (!$sh->GetFiles($site, $port, \@files)) {
+ 				info($k, $site, 'SiteHandler::GetFiles() failed for ' . $site);
+ 				next;
+ 			}
+ 		}
++                elsif (!$sh)
++		{
++			print "No \n"
++				unless ($settings{quiet});
++		}
+ 		elsif ($site->scheme eq 'ftp')
+ 		{
+ 			my $ftp;
+@@ -713,7 +721,8 @@ sub VersionCheck
+ 
+ 				# Got a response which wasn't HTTP 4xx -> bail out
+ 				if ($response->is_success && $response->status_line !~ /^4/) {
+-					info($k, $site, 'Not doing any guessing; site is lieing to us.');
++					print "URL: $url\n";
++					info($k, $site, "Skip guessing: Response not 4xx to a file that shouldnt exist (". $response->status_line .")");
+ 					$sths->{sitedata_initliecount}->execute($sitedata->{host})
+ 						unless($settings{precious_data});
+ 					next;
+@@ -797,7 +806,7 @@ sub VersionCheck
+ 							$new_found = 1;
+ 							last;
+ 						} else {
+-							info($k, $site, "Guess failed $port->{ver} -> $guess_v");
++							info($k, $site, "Guess failed $port->{ver} -> $guess_v (". $response->status_line .")");
+ 						}
+ 
+ 						last if ($new_found);
diff --git a/ports-mgmt/portscout/pkg-plist b/ports-mgmt/portscout/pkg-plist
index 346219edded8..d8d7e78f8c8c 100644
--- a/ports-mgmt/portscout/pkg-plist
+++ b/ports-mgmt/portscout/pkg-plist
@@ -14,19 +14,21 @@ man/man1/portscout.1.gz
 %%DATADIR%%/sql/sqlite_init.sql
 %%DATADIR%%/sql/sqlite_destroy.sql
 %%DATADIR%%/sql/sqlite_upgrade_0.8_to_0.8.1.sql
-%%SITE_PERL%%/Portscout/SQL/SQLite.pm
-%%SITE_PERL%%/Portscout/SQL/Pg.pm
-%%SITE_PERL%%/Portscout/SiteHandler/SourceForge.pm
+%%SITE_PERL%%/Portscout.pm
+%%SITE_PERL%%/Portscout/API.pm
+%%SITE_PERL%%/Portscout/Config.pm
+%%SITE_PERL%%/Portscout/Const.pm
+%%SITE_PERL%%/Portscout/DataSrc.pm
 %%SITE_PERL%%/Portscout/DataSrc/Ports.pm
 %%SITE_PERL%%/Portscout/DataSrc/XML.pm
-%%SITE_PERL%%/Portscout/SiteHandler.pm
-%%SITE_PERL%%/Portscout/SQL.pm
 %%SITE_PERL%%/Portscout/Make.pm
-%%SITE_PERL%%/Portscout/Const.pm
-%%SITE_PERL%%/Portscout/DataSrc.pm
+%%SITE_PERL%%/Portscout/SQL.pm
+%%SITE_PERL%%/Portscout/SQL/Pg.pm
+%%SITE_PERL%%/Portscout/SQL/SQLite.pm
+%%SITE_PERL%%/Portscout/SiteHandler.pm
+%%SITE_PERL%%/Portscout/SiteHandler/GitHub.pm
+%%SITE_PERL%%/Portscout/SiteHandler/PyPI.pm
+%%SITE_PERL%%/Portscout/SiteHandler/SourceForge.pm
 %%SITE_PERL%%/Portscout/Template.pm
 %%SITE_PERL%%/Portscout/Util.pm
-%%SITE_PERL%%/Portscout/API.pm
-%%SITE_PERL%%/Portscout/Config.pm
-%%SITE_PERL%%/Portscout.pm
 %%SITE_PERL%%/portscout.pod