You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/hooks.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,7 @@
1
1
# Hooks
2
2
3
3
::: tip
4
-
Although all are not applicable, [common FeathersJS hooks](https://auk.docs.feathersjs.com/api/hooks-common.html) are exposed in addition to krawler hooks and can be used in jobs, e.g. you can add `disallow: 'external'` to avoid exposing some services when deploying as a web app.
4
+
Although all are not applicable, [common FeathersJS hooks](https://auk.docs.feathersjs.com/api/hooks-common.html) are exposed in addition to krawler hooks and can be used in jobs, e.g. you can add `disallow: 'external'` to avoid exposing some services when deploying as a web app.
5
5
:::
6
6
7
7
## Common options
@@ -17,7 +17,7 @@ Due to templating restricted to string output any ISO date string or comparison
17
17
Matching is for instance useful when you'd like to apply a hook to only a subset of your tasks, e.g. all the CSV files but not the JSON files.
18
18
19
19
Fault tolerance is for instance useful when you use unreliable data sources and you don't want the job to stop when some requests fail.
20
-
:::
20
+
:::
21
21
22
22
All input/output hooks and store hooks manipulating items, i.e. reading/writing/transforming/removing data in a store like `readJson`, `writeJson` or `gzipToStore`, can have the following options:
23
23
***storePath**: property path where to read the store to be used on the hook object or params, defaults to `data.store`
@@ -180,7 +180,7 @@ Pull a docker image. Hook options are the following:
180
180
Run a docker container. Hook options are the following:
181
181
***clientPath**: property path where to retrieve the client object, defaults to `client`
182
182
* any options supported by dockerode for [container creation](https://github.com/apocas/dockerode#manipulating-a-container)
183
-
183
+
184
184
::: tip
185
185
`Cmd` and `Env` options can be templates, learn more about [templating](https://lodash.com/docs/4.17.4#template)
186
186
:::
@@ -190,7 +190,7 @@ Run a docker container. Hook options are the following:
190
190
Create a docker service on a Swarm cluster. Hook options are the following:
191
191
***clientPath**: property path where to retrieve the client object, defaults to `client`
192
192
* any options supported by dockerode for [service creation](https://github.com/apocas/dockerode#documentation)
193
-
193
+
194
194
::: tip
195
195
Options can be templates, learn more about [templating](https://lodash.com/docs/4.17.4#template)
196
196
:::
@@ -354,8 +354,8 @@ fetchIMAPMessages: {
354
354
mailbox:'INBOX',
355
355
range: { seen:false },
356
356
query: { uid:true },
357
-
clientPath:'taskTemplate.imapClient',
358
-
dataPath:'data.messages'
357
+
clientPath:'taskTemplate.imapClient',
358
+
dataPath:'data.messages'
359
359
}
360
360
```
361
361
### downloadIMAPAttachments
@@ -558,12 +558,12 @@ Read a KML from an input stream/store and convert it to in-memory JSON values, h
558
558
### connectMongo(options)
559
559
560
560
Connect to a MongoDB database. The [connection options](http://mongodb.github.io/node-mongodb-native/3.0/api/MongoClient.html#.connect) of the client are defined in the hook options plus:
***url**: MongoDB [URI connection string](https://docs.mongodb.com/manual/reference/connection-string/). This field can be a template with item as context, learn more about [templating](https://lodash.com/docs/4.17.4#template)
562
562
***dbName**: the name of the DB to connect to
563
563
***clientPath**: property path where to store the client object to be used by the MongoDB hooks, defaults to `client`
564
564
565
565
::: tip
566
-
Since **Krawler** relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option `useNewUrlParser: true` when connecting to the database.
566
+
Since **Krawler** relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option `useNewUrlParser: true` when connecting to the database.
567
567
:::
568
568
569
569
### disconnectMongo(options)
@@ -625,7 +625,7 @@ Inserts JSON into an existing collection (uses [insertOne](https://docs.mongodb.
625
625
***clientPath**: property path where to retrieve the client object, defaults to `client`
626
626
***transform**: perform transformation using these options before write, see description in [transformJson](./hooks.md#transformjson-options)
627
627
* any option supported by `options` argument of the [bulkWrite](https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/) function.
628
-
628
+
629
629
::: tip
630
630
If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.
631
631
:::
@@ -642,7 +642,7 @@ Updates JSON into an existing collection (uses [updateOne](https://docs.mongodb.
642
642
***filter/upsert/hint**: corresponding option for `updateOne` operation, filter fields can be templates, learn more about [templating](https://lodash.com/docs/4.17.4#template)
643
643
***excludedProperties**: array of property names to be excluded from automated type conversion after templating, useful if you have a number-like string (eg '81') that you don't want to convert automatically into a number
644
644
* any option supported by `options` argument of the [bulkWrite](https://docs.mongodb.com/manual/reference/method/db.collection.bulkWrite/) function.
645
-
645
+
646
646
::: tip
647
647
If the input data is a GeoJSON collection the array of features will be updated into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.
648
648
:::
@@ -669,7 +669,7 @@ Creates an [aggregation pipeline](https://docs.mongodb.com/manual/aggregation/#a
669
669
***skipAllConvert** allows to skip all conversions
670
670
* other options are passed as is
671
671
* any option supported by `options` argument of the [aggregate](https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/) function.
672
-
672
+
673
673
::: tip
674
674
If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.
675
675
:::
@@ -719,7 +719,7 @@ Connect to a Feathers API. The [connection options](https://docs.feathersjs.com/
719
719
***clientPath**: property path where to store the client object to be used by the Feathers hooks, defaults to `client`
720
720
721
721
::: tip
722
-
**Krawler** uses the version 5 of the Feathers client.
722
+
**Krawler** uses the version 5 of the Feathers client.
723
723
:::
724
724
725
725
### disconnectFeathers(options)
@@ -809,7 +809,7 @@ Also, this hook allows you to use the same environment variables as [node-postgr
809
809
*`PGPORT=5432`
810
810
*`PGDATABASE=database`
811
811
812
-
Finaly and for some security reason, it is highly recommended to combine both ways such as in the following example:
812
+
Finaly and for some security reason, it is highly recommended to combine both ways such as in the following example:
813
813
```
814
814
connectPG: {
815
815
user: process.env.PG_USER,
@@ -950,7 +950,7 @@ Untar files or directories using [node-tar](https://github.com/npm/node-tar), ho
950
950
951
951
Run a system command. Hook options are the following:
952
952
***command**: the template of the command to be run with the hook object as context (could be an array commands for a sequence)
953
-
***spawn**: `true` to use `child_process.spawn` instead of `child_process.exec` (default) to run the command(s), in that case a command is given as an array of args instead of a single string
953
+
***spawn**: `true` to use `child_process.spawn` instead of `child_process.exec` (default) to run the command(s), in that case a command is given as an array of args instead of a single string
954
954
***stdout**: boolean indicating if stdout is logged and stored in the hook object
955
955
***stderr**: boolean indicating if stderr is logged and stored in the hook object
0 commit comments