Skip to content

Increase testability by using a docker instance #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Increase testability by using a docker instance #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@

### Perl ###
!Build/
.last_cover_stats
/META.yml
/META.json
/MYMETA.*
*.o
*.pm.tdy
*.bs

# ExtUtils::MakeMaker
/blib/
/_eumm/
/*.gz
/Makefile
/Makefile.old
/MANIFEST.bak
/pm_to_blib
/*.zip
106 changes: 106 additions & 0 deletions docker/functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Shell functions for running docker containers -*- bash -*-

startContainer() {
port=$1
containerName=$2

docker run -d -p $port:80 -e LC_ALL=C.UTF-8 $containerName
}

isContainerUp() {
container=$1

docker logs $container 2>&1 | grep -q apache2.-D.FOREGROUND
echo $?
}

getContainerReady() {
port=$1
containerName=$2

container=$(startContainer $port $containerName)
ready=$(isContainerUp $container)

while [ "$ready" -ne 0 ] ; do
ready=$(isContainerUp $container)

sleep 1
done
echo $container
}

setupMW() {
container=$1
wikiadmin=$2
wikipass=$3

docker exec $container php /var/www/html/maintenance/install.php \
--dbtype=sqlite --dbpath=/tmp --pass=$wikipass \
--skins=Vector --scriptpath= core $wikiadmin | \
grep -q 'MediaWiki has been successfully installed'
if [ $? -ne 0 ]; then
echo Trouble installing MediaWiki in container! 1>&2
teardownContainer $container
exit 2
fi
docker exec $container sh -c \
'chown www-data /tmp/*.sqlite /var/www/html/images \
/var/www/html/cache'
# Turn on uploads for the tests
docker exec $container sh -c \
'echo "\$wgEnableUploads = true;" >> \
/var/www/html/LocalSettings.php'
# Allow i18n caching to disk
docker exec $container sh -c \
'echo "\$wgCacheDirectory = \"\$IP/cache\";" >> \
/var/www/html/LocalSettings.php'
# Turn off most caching
docker exec $container sh -c \
'echo "\$wgMainCacheType = CACHE_NONE;" >> \
/var/www/html/LocalSettings.php'
# Run a lot of jobs each time
docker exec $container sh -c \
'echo "\$wgRunJobsAsync = false;" >> \
/var/www/html/LocalSettings.php'
docker exec $container sh -c \
'echo "\$wgJobRunRate = 10;" >> \
/var/www/html/LocalSettings.php'
# Debug logs for troubleshooting
docker exec $container sh -c \
'echo "\$wgDebugLogFile = \"/tmp/debug.log\";" >> \
/var/www/html/LocalSettings.php'
}

teardownContainer() {
container=$1

result=$(docker kill $container)
if [ "$result" != "$container" ]; then
echo Some problem with the kill?
exit 3
fi

result=$(docker rm $container)
if [ "$result" != "$container" ]; then
echo Some problem with removal?
exit 4
fi
}

# From https://unix.stackexchange.com/a/358101
getFreePort() {
netstat -aln | awk '
$6 == "LISTEN" {
if ($4 ~ "[.:][0-9]+$") {
split($4, a, /[:.]/);
port = a[length(a)];
p[port] = 1
}
}
END {
for (i = 3000; i < 65000 && p[i]; i++){};
if (i == 65000) {exit 1};
print i
}
'
}
40 changes: 40 additions & 0 deletions docker/run-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/bash

. docker/functions.sh

WIKI_DIAGNOSTICS=${WIKI_DIAGNOSTICS:-0}
export WIKI_DIAGNOSTICS

WIKI_PASS=${WIKI_PASS:-pass123456}
export WIKI_PASS

WIKI_ADMIN=${WIKI_ADMIN:-admin}
export WIKI_ADMIN

WIKI_PORT=${WIKI_PORT:-$(getFreePort)}

WIKI_API_URL=${WIKI_API_URL:-http://localhost:$WIKI_PORT/api.php}
export WIKI_API_URL

# 1.27 does not work due to a change made in install.php
WIKI_IMAGE=${WIKI_IMAGE:-mediawiki:latest}

if [ -z "$WIKI_PORT" ]; then
echo Could not find a free port to use.
exit 10
fi

echo -n "Preparing container to listen on $WIKI_PORT... "
container=$(getContainerReady $WIKI_PORT $WIKI_IMAGE)
echo up as $container.

echo -n "Setting up SQLite-based MW... "
setupMW $container $WIKI_ADMIN $WIKI_PASS
echo done.

echo Running tests...
make; make test || echo FAILURES, but killing container anyway

echo -n "Tearing down container... "
teardownContainer $container
echo done.
75 changes: 58 additions & 17 deletions lib/MediaWiki/API.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ use URI::Escape;
use Encode;
use JSON;
use Carp;
use Data::Dumper;

# just for debugging the module
# use Data::Dumper;
# use Devel::Peek;

use constant {
Expand Down Expand Up @@ -118,6 +118,8 @@ Configuration options are

=item * no_proxy = Boolean; Set to 1 to Disable use of any proxy set in the environment. Note by default if you have proxy environment variables set, then the module will attempt to use them. This feature was added at version 0.29. Versions below this ignore any proxy settings, but you can set this yourself by doing MediaWiki::API->{ua}->env_proxy() after creating a new instance of the API class. More information about env_proxy can be found at http://search.cpan.org/~gaas/libwww-perl-5.834/lib/LWP/UserAgent.pm#Proxy_attributes

=item * diagnostics = Boolean; Diagnostics will be printed to STDERR

=back

An example for the on_error configuration could be something like:
Expand Down Expand Up @@ -197,6 +199,7 @@ sub new {
$self->{error}->{details} = '';
$self->{error}->{stacktrace} = '';

$self->{diagnostics} = $config->{diagnostics};
bless ($self, $class);
return $self;
}
Expand All @@ -221,9 +224,26 @@ sub _get_config_defaults {
'paraminfo' => 1
};

$config{diagnostics} = 0;

return \%config;
}

sub _diag {
my ( $self, $msg ) = @_;

if ( $self->{diagnostics} ) {
for ($msg) {
s{\n}{\n# }gsmx;
s{(Doing|Returning).[\$]VAR1.=.}{$1 }smx;
s{;$}{}smx;
}

warn "# $msg\n";
}
return;
}

=head2 MediaWiki::API->login( $query_hashref )

Logs in to a MediaWiki. Parameters are those used by the MediaWiki API (http://www.mediawiki.org/wiki/API:Login). Returns a hashref with some login details, or undef on login failure. If Mediawiki sends requests a LoginToken the login is attempted again, but with the token sent from the initial login. Errors are stored in MediaWiki::API->{error}->{code} and MediaWiki::API->{error}->{details}.
Expand Down Expand Up @@ -344,6 +364,7 @@ sub api {

my $retries = $self->{config}->{retries};
my $maxlagretries = 1;
$self->_diag( 'Doing ' . Dumper($query) );

$self->_encode_hashref_utf8($query, $options->{skip_encoding});
$query->{maxlag} = $self->{config}->{max_lag} if defined $self->{config}->{max_lag};
Expand All @@ -364,32 +385,40 @@ sub api {

# if we are already retrying, then wait the specified delay
if ( $try > 0 ) {
$self->_diag('Sleeping before retry.');
sleep $self->{config}->{retry_delay};
}

my $response;
my %headers;
# if we are using the get method ($querystring is set above)
if ( $querystring ) {
$self->_diag("GETting $querystring");
$response = $self->{ua}->get( $self->{config}->{api_url} . $querystring, %headers );
} else {
$headers{'content-type'} = 'form-data' if $query->{action} eq 'upload' || $query->{action} eq 'import';
if ( $query->{action} eq 'upload' || $query->{action} eq 'import' ) {
$self->_diag("'{$query->{action}}' uses form-data header");
$headers{'content-type'} = 'form-data';
}
$response = $self->{ua}->post( $self->{config}->{api_url}, $query, %headers );
}
$self->{response} = $response;

# if the request was successful then check the returned content and decode.
if ( $response->is_success ) {

$self->_diag('Successful request.');
my $decontent = $response->decoded_content( charset => 'none' );

if ( ! defined $decontent ) {
$self->_diag('Unable to decode HTTP content body');
return $self->_error(ERR_HTTP,"Unable to decode content returned by $self->{config}->{api_url} - Unknown content encoding?")
if ( $try == $retries );
next;
}
}

if ( length $decontent == 0 ) {
$self->_diag('Zero-length content.');
return $self->_error(ERR_HTTP,"$self->{config}->{api_url} returned a zero length string")
if ( $try == $retries );
next;
Expand All @@ -401,10 +430,14 @@ sub api {
};

if ( $@) {
$self->_diag("Failed to decode content: $decontent");
# an error occurred, so we check if we need to retry and continue
my $error = $@;
return $self->_error(ERR_HTTP,"Failed to decode JSON returned by $self->{config}->{api_url}\nDecoding Error:\n$error\nReturned Data:\n$decontent")
if ( $try == $retries );
if ( $try == $retries ) {
$self->_diag(
'Retry limit reached without a successful response.');
return $self->_error(ERR_HTTP,"Failed to decode JSON returned by $self->{config}->{api_url}\nDecoding Error:\n$error\nReturned Data:\n$decontent")
}
next;
} else {
# no error so we want out of the retry loop
Expand All @@ -417,32 +450,40 @@ sub api {
return $self->_error(ERR_HTTP, $response->status_line . " : error occurred when accessing $self->{config}->{api_url} after " . ($try+1) . " attempt(s)")
if ( $try == $retries );
next;
}
}

}

return $self->_error(ERR_API,"API has returned an empty array reference. Please check your parameters") if ( ref($ref) eq 'ARRAY' && scalar @{$ref} == 0);
if ( ref($ref) eq 'ARRAY' && scalar @{$ref} == 0) {
$self->_diag('Empty array response.');
return $self->_error(ERR_API,"API has returned an empty array reference. Please check your parameters")
}

# check lag and wait
if (ref($ref) eq 'HASH' && exists $ref->{error} && $ref->{error}->{code} eq 'maxlag' ) {
if ($maxlagretries == $self->{config}->{max_lag_retries}) {
return $self->_error(ERR_API,"Server has reported lag above the configured max_lag value of " . $self->{config}->{max_lag} . " value after " . $self->{config}->{max_lag_retries} . " attempt(s). Last reported lag was - ". $ref->{'error'}->{'info'})
} else {
sleep $self->{config}->{max_lag_delay};
$maxlagretries++ if $maxlagretries < $self->{config}->{max_lag_retries};
# redo the request
next;
}
if ($maxlagretries == $self->{config}->{max_lag_retries}) {
$self->_diag('Reached limit for too much lag time');
return $self->_error(ERR_API,"Server has reported lag above the configured max_lag value of " . $self->{config}->{max_lag} . " value after " . $self->{config}->{max_lag_retries} . " attempt(s). Last reported lag was - ". $ref->{'error'}->{'info'})
} else {
sleep $self->{config}->{max_lag_delay};
$maxlagretries++ if $maxlagretries < $self->{config}->{max_lag_retries};

$self->_diag('Too much lag time, retrying');
# redo the request
next;
}
}

# if we got this far, then we have a hashref from the api and we want out of the while loop
last;

}

return $self->_error(ERR_API,$ref->{error}->{code} . ": " . $ref->{error}->{info} ) if ( ref($ref) eq 'HASH' && exists $ref->{error} );

if ( ref($ref) eq 'HASH' && exists $ref->{error} ) {
$self->_diag( 'Error from server: ' . $ref->{error}->{info} );
} else {
$self->_diag( 'Returning ' . Dumper($ref) );
}
return $ref;
}

Expand Down
Loading