Skip to content

Commit 9fdfcc4

Browse files
committed
Service
* Added service that parses data csv and metadata csv * Using the new data structures the service parses the data and creats a json * Added tests to the service Updated import script for v2 * Removed all files that parse mongodb and removed dependency * Added service that handles incoming csv data for metadata and participant csv data * Service also has a method to convert data to json * Added api request with a configuration for url Add requests * Added requests dependency Updates to script to add hash collection * Added hash collection
1 parent 0ce66a4 commit 9fdfcc4

File tree

20 files changed

+357
-879
lines changed

20 files changed

+357
-879
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22
*.pyc
33
*.egg-info
44
docs/build
5+
.vscode

README.md

Lines changed: 12 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,40 @@
1-
DPimport: A command line glob importer for DPdash
2-
=================================================
1+
# DPimport: A command line glob importer for DPdash
2+
33
DPimport is a command line tool for importing files into DPdash using a
4-
simple [`glob`](https://en.wikipedia.org/wiki/Glob_(programming)) expression.
4+
simple [`glob`](<https://en.wikipedia.org/wiki/Glob_(programming)>) expression.
55

66
## Table of contents
7+
78
1. [Installation](#installation)
89
2. [Configuration](#configuration)
910
3. [Usage](#usage)
10-
4. [MongoDB](#mongodb)
1111

1212
## Installation
13+
1314
Just use `pip`
1415

1516
```bash
1617
pip install https://github.com/AMP-SCZ/dpimport.git
1718
```
1819

1920
## Configuration
21+
2022
DPimport requires a configuration file in YAML format, passed as a command
21-
line argument with `-c|--config`, for establishing a MongoDB database
22-
connection. You will find an example configuration file in the `examples`
23+
line argument with `-c|--config`, for establishing a MongoDB database
24+
connection. You will find an example configuration file in the `examples`
2325
directory within this repository.
2426

2527
## Usage
28+
2629
The main command line tool is `import.py`. You can use this tool to import any
27-
DPdash-compatible CSV files or metadata files using the direct path to a file
30+
DPdash-compatible CSV files or metadata files using the direct path to a file
2831
or a glob expression (use single quotes to avoid shell expansion)
2932

3033
```bash
3134
import.py -c config.yml '/PHOENIX/GENERAL/STUDY_A/SUB_001/DATA_TYPE/processed/*.csv'
3235
import.py -c config.yml '/PHOENIX/GENERAL/STUDY_A/SUB_001/DATA_TYPE/processed/*.csv' -n 8
3336
```
3437

35-
`-n 8` is for parallelly importing 8 files. The default is `-n 1`.
36-
37-
3838
You may also now use the `**` recursive glob expression, for example:
3939

4040
```bash
@@ -55,18 +55,9 @@ and so on.
5555

5656
`directory/*/*.csv` matches only `directory/[subdirectory]/[filename].csv`. With a [recursive glob pattern](https://docs.python.org/3/library/glob.html#glob.glob), `directory/**/*.csv` will additionally match:
5757

58-
* `directory/[filename].csv` (no subdirectory)
59-
* `directory/[subdirectory1]/[subdirectory2]/[filename].csv` (sub-subdirectory)
58+
- `directory/[filename].csv` (no subdirectory)
59+
- `directory/[subdirectory1]/[subdirectory2]/[filename].csv` (sub-subdirectory)
6060

6161
and so on, for as many levels deep as exist in the directory tree.
6262

6363
</details>
64-
65-
66-
67-
## MongoDB
68-
69-
This tool requires MongoDB to be running and accessible with the credentials you
70-
supply in the `config.yml` file. For tips on MongoDB as it is used in DPdash and DPimport,
71-
see [the DPdash wiki](https://github.com/PREDICT-DPACC/dpdash/wiki/MongoDB-Tips).
72-

dpimport/__init__.py

Lines changed: 0 additions & 104 deletions
This file was deleted.

dpimport/__version__.py

Lines changed: 0 additions & 6 deletions
This file was deleted.

dpimport/database/__init__.py

Lines changed: 0 additions & 102 deletions
This file was deleted.

0 commit comments

Comments
 (0)