Skip to content

Metadata test harness #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 102 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
4add0cb
Testing new quoting fixes for DB creds.
valayDave Sep 2, 2021
f0caa22
Fixed Docker image based go image.
valayDave Sep 3, 2021
e656dd9
reverting old commit
valayDave Sep 3, 2021
1e13c4e
Creating shell of a mini test harness with docker
valayDave Sep 3, 2021
98ca3d2
added main command in env builder.
valayDave Sep 3, 2021
c511caf
Added comments.
valayDave Sep 3, 2021
aa10b7f
import tweek ?
valayDave Sep 3, 2021
6f71d2c
fix.
valayDave Sep 3, 2021
4826bab
Fixing bug >?
valayDave Sep 3, 2021
221848f
bug fix
valayDave Sep 3, 2021
858edfa
fix
valayDave Sep 3, 2021
3d895f8
Added Fixes to test harness shell.
valayDave Sep 3, 2021
fd54555
Added logs and comments.
valayDave Sep 3, 2021
2f8bf2e
Got test harness kinda working
valayDave Sep 4, 2021
8d20390
Testing buffer name changes
valayDave Sep 4, 2021
729f141
Fixing bugs
valayDave Sep 4, 2021
7581109
Fixing temp versioning arguement.
valayDave Sep 4, 2021
a83779b
Probable bug fix for logs ?
valayDave Sep 4, 2021
ea157ab
added log
valayDave Sep 4, 2021
3e369b0
prettyinh logs
valayDave Sep 4, 2021
cff02b4
Dependency baking.
valayDave Sep 4, 2021
0ec7da0
Fixing more requirements.
valayDave Sep 4, 2021
1716c2c
temporary fix.
valayDave Sep 4, 2021
1ea130b
added db port tweeks.
valayDave Sep 4, 2021
6aac834
adding meaningless log to debug
valayDave Sep 4, 2021
a95228b
added logs
valayDave Sep 4, 2021
c84aaff
logging fix and tweek
valayDave Sep 4, 2021
94e1e0e
logs and fixes.
valayDave Sep 4, 2021
64d0047
trying bug fix.
valayDave Sep 4, 2021
f4edac5
Increasing timeouts to ip resolution
valayDave Sep 4, 2021
59a685c
Fixing ip resolution
valayDave Sep 4, 2021
5ab06de
logging bug fix
valayDave Sep 4, 2021
2b45c3d
adding sleep to fix init?
valayDave Sep 4, 2021
e7cb452
bug fix?
valayDave Sep 4, 2021
11b6e9a
Hard baking deps
valayDave Sep 4, 2021
9ebdfaa
tweek in dockerfile
valayDave Sep 4, 2021
aaf5331
checking with changed image
valayDave Sep 4, 2021
2fd0dab
fix
valayDave Sep 4, 2021
4846490
fixing bug
valayDave Sep 4, 2021
3564103
Removing pointless logs.
valayDave Sep 4, 2021
e847b52
Tweeking fixes.
valayDave Sep 4, 2021
06ceb54
tweek
valayDave Sep 4, 2021
dca6e0d
Tweek.
valayDave Sep 4, 2021
64ea2f4
adding logs + hygenie refactor
valayDave Sep 4, 2021
bbaf0a4
bug fix
valayDave Sep 4, 2021
0ac2726
hygenie refactor;
valayDave Sep 4, 2021
4c9c613
bug fix
valayDave Sep 4, 2021
a134e84
minimizing timeouts
valayDave Sep 4, 2021
57b3e4f
Adding logs to return responses
valayDave Sep 4, 2021
2053f47
increasing timeout between flows.
valayDave Sep 4, 2021
d5ea731
changing to MP pool to test
valayDave Sep 4, 2021
511aec4
testing tweek
valayDave Sep 4, 2021
b2516f0
rolling back change
valayDave Sep 4, 2021
2eca2e4
testing mp pool
valayDave Sep 4, 2021
db03f19
adding job lib to scheduler
valayDave Sep 4, 2021
9ef43f0
reducing concurrency
valayDave Sep 4, 2021
e271664
changing parallel to serial
valayDave Sep 4, 2021
c466a88
added error handle
valayDave Sep 4, 2021
cee2c43
logging
valayDave Sep 4, 2021
3e2d213
tsting tweek password
valayDave Sep 4, 2021
31e52cc
Refactored code to decouple execution
valayDave Sep 4, 2021
bfc1517
bug fix.
valayDave Sep 4, 2021
b257e7f
adding dummy default password
valayDave Sep 4, 2021
e4d5d9a
uncoupling goose version for test
valayDave Sep 4, 2021
f1f48a8
Unpinning dependences to check effects on ttests.
valayDave Sep 4, 2021
e116f32
testing psycopg2 make dsn fn
valayDave Sep 6, 2021
f019ec1
fixiing bug
valayDave Sep 6, 2021
3a34f64
Reverting requirements to version freeze.
valayDave Sep 6, 2021
a1a6475
Decouping docker filee
valayDave Sep 6, 2021
916bfb0
Added version spec for quick tests
valayDave Sep 6, 2021
f169aa2
Tweeking password for test.
valayDave Sep 6, 2021
d252c3e
bug fix
valayDave Sep 6, 2021
c2d1d59
Adding log printing in tests.
valayDave Sep 6, 2021
34cb302
Testing new logging change
valayDave Sep 6, 2021
ee445a5
adding hygenie checks for replaying env
valayDave Sep 6, 2021
95378aa
added to do.
valayDave Sep 6, 2021
b27383f
Following changes:
valayDave Sep 7, 2021
31150ac
Dumb log
valayDave Sep 7, 2021
d7e1dca
dummy log
valayDave Sep 7, 2021
026a7a2
Fixing bug with port exposure.
valayDave Sep 7, 2021
add4e8a
dummy print statement.
valayDave Sep 7, 2021
fbaf173
Dummy log
valayDave Sep 7, 2021
b66e1f7
dummy log
valayDave Sep 7, 2021
2c19ae7
Possible fix to Shell problem ?
valayDave Sep 7, 2021
4e84ffe
Unbundling Goose Dep
valayDave Sep 7, 2021
b0dac42
Moving to unpinned deps in requirements
valayDave Sep 7, 2021
82b1e38
Reverting useless log
valayDave Sep 7, 2021
8e4836a
Added changes relating to connectionstring.
valayDave Sep 8, 2021
1a9a3fa
fixing bug in conn str ?
valayDave Sep 8, 2021
3198ff2
Adding more complexity to test
valayDave Sep 8, 2021
1626fc0
fixed imports;
valayDave Sep 8, 2021
b641cb1
removing pointless imports;
valayDave Sep 8, 2021
71877ab
added dummy logs
valayDave Sep 8, 2021
30d2a99
Tweeking connection string.
valayDave Sep 9, 2021
34f6ea5
Refactored goose cmd code for shlex.
valayDave Sep 9, 2021
4ecbdca
Fixing import error .
valayDave Sep 9, 2021
cf49f1f
Removed quotes.
valayDave Sep 9, 2021
291d16e
Refactored Exception to make it simple
valayDave Sep 14, 2021
9abaf24
Merge branch 'master' into metadata-test-harness
valayDave Oct 12, 2021
e9bc2d9
removing dead code + fixing warnings.
valayDave Oct 12, 2021
6e4a9a2
Added MF versions.
valayDave Oct 12, 2021
ba5f58d
Tweek to the test to ensure run 0 is captured.
valayDave Oct 12, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Added Fixes to test harness shell.
- todo: integrating tests.
valayDave committed Sep 3, 2021
commit 3d895f849fc1ee03c035dce68feba7b6097d862e
32 changes: 26 additions & 6 deletions docker-tests/env_buider.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import abc
import docker
import click
from docker.errors import BuildError, NotFound, APIError
import time
from versioned_tests import EnvConfig,MFTestRunner
@@ -38,11 +39,15 @@ def __init__(self,\
database_password='password',
database_user='metaflow',
database_port = 5432,
logger=None,
image_name = 'metaflow_metadata_service',
network_name='postgres-network',\
image_build_path='../',\
docker_file_path=os.path.abspath('../Dockerfile')) -> None:

self._docker = docker.DockerClient(base_url='unix://var/run/docker.sock')

self._logger = logger if logger is not None else lambda *args:print(*args)

# Network Related Properties
self._network = None
@@ -62,12 +67,16 @@ def __init__(self,\
self._image_name = image_name # Image of MD Service
self._metadataservice_container = None
self._metadata_image = None
self._image_build_path = image_build_path
self._metadataservice_name = 'testharness-metadataservice'

def lifecycle(self):
# Create the network and the image.
self._logger('Creating New Environment',fg='green')
self._create_enivornment()
self._logger('Environment Created',fg='green')
self._run_tests()
self._logger('Tearing down environment',fg='green')
self._teardown_environment()

def _run_tests(self):
@@ -78,20 +87,24 @@ def _teardown_environment(self):
self._metadataservice_container,
self._database_container
]
self._logger('Stopping all containers',fg='blue')
# first stop all containers
for container in container_set:
container.stop(timeout=10)
container.reload()

self._logger('Removing all containers',fg='blue')
# Then remove all the containers
for container in container_set:
container.remove()

self._logger('Removing Network',fg='blue')
# Remove the network
self._network.remove()


self._logger('Removing Docker Images',fg='blue')
# remove the images.
self._docker.images.remove(self._metadata_image.name)
self._docker.images.remove(self._metadata_image.id)


def _db_env_vars(self):
@@ -133,32 +146,37 @@ def _mdservice_ports(self):
}

def _create_enivornment(self):
self._logger('Creating a network',fg='blue')
self._network = self._find_network()

if self._network is None:
self._network = self._create_network()

# Build the image of the Metadata service.
self._logger('Building Metadata Image',fg='blue')
self._metadata_image = self._build_mdservice_image()

self._logger('Creating Postgres Docker Container',fg='blue')
# Create the Postgres container
self._database_container = self._docker.containers.run(POSTGRES_IMAGE,\
detach=True,\
name=self._database_container_name,\
environment=self._db_env_vars(),\
network=self._network_name,)

self._logger('Creating Metadata Service Container',fg='blue')
# Create the metadata service container
self._metadataservice_container = self._docker.containers.run(self._image_name,\
detach=True,\
envionment=self._mdcontainer_env_vars(),\
environment=self._mdcontainer_env_vars(),\
ports=self._mdservice_ports(),\
)


def _build_mdservice_image(self):
dockerfileobj = open(self._docker_file_path)
image,log_generator = self._docker.images.build(path=self._docker_file_path,fileobj=dockerfileobj,tag=self._image_name,)
image,log_generator = self._docker.images.build(path=self._image_build_path,\
dockerfile=self._docker_file_path,\
tag=self._image_name,)
return image

def _find_network(self):
@@ -173,7 +191,9 @@ def _create_network(self):
return self._docker.networks.create(self._network_name)

def run_tests():
test_runner = DockerTestEnvironment()
test_runner = DockerTestEnvironment(
logger=click.secho
)
test_runner.lifecycle()

if __name__ == '__main__':