diff --git a/.gitignore b/.gitignore index bb528281e..1b14da5ff 100644 --- a/.gitignore +++ b/.gitignore @@ -7,7 +7,7 @@ yarn-error.log* lerna-debug.log* # Generated jsdocs -docs/ +packages/**/docs/ # Working folders tmp/ diff --git a/docs/best-practice.md b/docs/best-practice.md new file mode 100644 index 000000000..87bd39334 --- /dev/null +++ b/docs/best-practice.md @@ -0,0 +1,33 @@ +style guide and best practice notes + +## Code Style + +* Use `camelCase` in variable and parameter names. This helps keep adaptors consistent. +* Try to avoid exposing the underlying implementation. +* * For example, if wrapping a function and the function exposes an options object, it's usually better to define your own options object and map it. +* * This lets us provide our own documention, enforce camelCase conventions (particularly when wrapping web services), and insulates us from changes in the underlying implementation. +* * It can also simplify an API to just expose options an end-user needs. For example we'd expect than an adaptor should deal with pagination automagically, so exporting page options is probably unhelpful +* * This isn't _always_ possible or sensible, particularly for large APIs, but should be considered. + +## Documentation + +All operations should be well documented in JSdoc annotations. + +It is usually best not to link to the underlying implementation's documentation - their context, requirements and expectations are very different to ours, and it may confuse our users. Better to keep users on our own docsite, with consistent style and navigation. + +## API Design + +A successful adaptor should have a clean, re-usable and easy to understand API design. + +There are two schools of thought for adaptor API design [and frankly we haven't quite settled on an answer yet]. + +1) An adaptor is a one-for-one wrapper around some backing web service or library. For each endpoint/function in the service, there should be an operation, with the same name, in the adaptor. +2) An adaptor is a high-level, user friendly, opinionated wrapper around a web service or library. It is not a mapping, but a high level abstraction. + +The approach you take my depend on your experience, and that of your users. If you know your users are experts in a particular rest interface, it probably makes sense just to expose that interface in an openfn-y way (so, a 1:1 mapping). + +On the other hand, if an API is particularly complex or heavy in boilerplate, it probably makes sense to provide some user-friendly, high level abstractions. + +A good approach is probably to do a little of both. Provide low-level helper functions to give access to the underlying implementaton (wrapping the `request` function is often the first step to a successful adaptor!), but then creating opinionated helper functions with specific tasks in mind. + +Also, when creating/editing adaptors, it's often best to have a goal in mind. Don't try and wrap an entire implementation - just do the bits you need for the job you (or your users) have in mind. \ No newline at end of file diff --git a/docs/changes.md b/docs/changes.md new file mode 100644 index 000000000..a3dae5c2b --- /dev/null +++ b/docs/changes.md @@ -0,0 +1,9 @@ +## Internal / Temporary docs + +Tracking changes in this repo at a high level + +This will feed into the PR. + +* Introduce the `operation` factory helper +* Introduce impl pattern +* Introduce integration testing \ No newline at end of file diff --git a/docs/creating-adaptors.md b/docs/creating-adaptors.md new file mode 100644 index 000000000..12011c2d7 --- /dev/null +++ b/docs/creating-adaptors.md @@ -0,0 +1,101 @@ +explain the basic structure of how to create an adaptor + +explain the adaptor/impl pattern + +## Folder structure + +In order to make it easier for developers to create and maintain adaptors, we adopt strong conventions around style and structure. + +An adaptor should have the following file structure: +``` +src + +- operations.js + +- impl.js + +- util.js + +- index.js + +- mock.js +``` + +`operations.js` is the most important file and every adaptor should have one. It's where your operations will live, along with your documentation. Anything you expect users to call from job code should live here. Keeping your operations in one place makes it easy for other developers and maintainers to see where the user-facing code is. + +`impl.js` is not required but encouraged. A useful pattern is to declare the interface to your operations in `operations.js`, but to add the actual implementation, the actual logic, in `impl.js`. Splitting each operation into two parts makes it easier to define unit tests (see the testing doc) + +`util.js` is another optional file. If you have helper functions to support your implementation, you can add them here. If you think they might be of use to job writers, you can export them in index (see below) + +`mock.js` contains mock code and data for your unit and integration tests. This is excluded from the release build and not published to npm. + +`index.js` declares the exports used by the final npm module. + +## Defining operations + +The common adaptor exposes a helper function to make it easier to define adaptors. + +Call the `operation` function with a single function argument. The function takes state as its first argument and whatever else you need. Assign the function to an exported const. + +``` +export const getPatient = operation((state, name, country, options) => { + // Implement the operation here +}) +``` + +If you prefer, you can stll define operations the "old" way, by creating your own factory wrapper. +``` +export function getPatient(name, country, options) { + return (state) => { + // Implement the operation here + } +} +``` +You'll still see this pattern used across the repo. + +## Implementing Operations + +Testing OpenFn adaptors has traditionally been a very hard problem. The factory pattern and the nature of the backend service makes it very hard to test code in isolation. + +The impl pattern helps this. + +You can cut the body of each operation out into an impl. The impl has almost the same structure, but it also accepts a library, client or exectutor function as an argument. This makes the implementation mockable. + +For example: + +``` +// operations.js +import client from 'my-library'; +import impl from './impl'; + +export const getPatient = operation((state, name, country, options) => { + impl.getPatient(state, client, name, country, options) +}) +``` + +This operation is really just a signature and declaration which calls out to an implementation function. Look closely at the impl signature though and you'll notice an extra argument: `client`. + +Our implementation might look like ths: +``` +// impl.js +export const getPatient = (state, client, name, country, options) => { + const result = await client.findPatient(name, { country, ...options }) + state.data = result.patients[0]; + return state; +} +``` + +Default values should be set in the outer operation wrapper. + +The implementation is easy to unit test because we can pass in a mock client object or function. Different adaptors have different needs but they ultimately will either call out to a webservice or another npm module. + +## Exports + +In order for anyone to use your adaptor, you need to export the correct functions in the index. + +Usually this is as simple as: +``` +export * from './operations` +``` +Which will make all operations available to user jobs. + +You may also want to export util functions to job code. It's good practice to namespace these so that it's clear to users what is an operation and what is a function. +``` +export * as util from './utils' +``` +Users can then do `util.convertPatient(patient)` in their job code. \ No newline at end of file diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 000000000..0b80e1e8e --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,40 @@ +Overview of adaptors and how they work + +key concepts - explain the words job, expression, workflow etc + +## Jobs + +A job is a unit of work, expressed via Javascript-like + +We sometimes use the word "expression" when referring to a job. Technically, the code written by end users (and executed by the OpenFn runtime) is the _expression_. A job is an expression plus a bunch of metadata about it, like which adaptor to use, what initial state to pass in, what options to use during execution. + +But mostly the terms are interchangable. + +An expression can be compiled or uncompiled. + +## DSL + +OpenFn code does not use Javascript. + +It actually uses a Javascript-like DSL (Domain Specific Language). This needs to be compiled into executable Javascript. + +The major differences between openfn code and Javascript are: +* The top level functions in the code are executed synchronously (in sequence), even if they contain asynchronous code +* OpenFn code does not contain import statements (although technically it can). These are compiled in. + +## Operations + +Users write openfn code by composing operations into a sequence of commands. + +Adaptors export operations - basically helper functions - to users. + +### Operations vs functions + +While an operation is of course a Javascript function, there are several differences between operationns and regular javascript functions. + +* An operation is a function that returns a function which in turn takes and returns a state object. +* The operations declared in an adaptor are _factories_ which return state-handling functions. +* Operations can be invoked at the top level of job code +* To call an operation within another function in job code, it has to be wrapped with state, ie `myOp(options)(state)`. We consider this to be an antipattern in job code - but it is occasionally neccessary + +In short, an operation is a function designed to run in an openfn job and handle state. \ No newline at end of file diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 000000000..a074ef30d --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,15 @@ +explain how unit tests and integration tests work + + +There are two types of test which serve different purposes; + +* Unit Tests are tightly scoped tests which execute specific functions in isolation. The smaller a unit of code is tested, the better the test. We would expect one unit test per operation (usually against the implementation, rather than the operation) +* Integration tests are broadly scoped tests against multiple operations. These look like actual job code and are executed against the actual Openfn Runtime + +## How to Unit Test + +TODO + +## How to Integration Test + +TODO \ No newline at end of file diff --git a/packages/common/ast.json b/packages/common/ast.json index 50aaf973b..534e56958 100644 --- a/packages/common/ast.json +++ b/packages/common/ast.json @@ -43,49 +43,6 @@ }, "valid": true }, - { - "name": "fn", - "params": [ - "func" - ], - "docs": { - "description": "Creates a custom step (or operation) for more flexible job writing.", - "tags": [ - { - "title": "public", - "description": null, - "type": null - }, - { - "title": "example", - "description": "fn(state => {\n // do some things to state\n return state;\n});" - }, - { - "title": "function", - "description": null, - "name": null - }, - { - "title": "param", - "description": "is the function", - "type": { - "type": "NameExpression", - "name": "Function" - }, - "name": "func" - }, - { - "title": "returns", - "description": null, - "type": { - "type": "NameExpression", - "name": "Operation" - } - } - ] - }, - "valid": true - }, { "name": "jsonValue", "params": [ @@ -139,110 +96,6 @@ }, "valid": true }, - { - "name": "sourceValue", - "params": [ - "path" - ], - "docs": { - "description": "Picks out a single value from source data.\nIf a JSONPath returns more than one value for the reference, the first\nitem will be returned.", - "tags": [ - { - "title": "public", - "description": null, - "type": null - }, - { - "title": "example", - "description": "sourceValue('$.key')" - }, - { - "title": "function", - "description": null, - "name": null - }, - { - "title": "param", - "description": "JSONPath referencing a point in `state`.", - "type": { - "type": "NameExpression", - "name": "String" - }, - "name": "path" - }, - { - "title": "returns", - "description": null, - "type": { - "type": "NameExpression", - "name": "Operation" - } - } - ] - }, - "valid": true - }, - { - "name": "source", - "params": [ - "path" - ], - "docs": { - "description": "Picks out a value from source data.\nWill return whatever JSONPath returns, which will always be an array.\nIf you need a single value use `sourceValue` instead.", - "tags": [ - { - "title": "public", - "description": null, - "type": null - }, - { - "title": "example", - "description": "source('$.key')" - }, - { - "title": "function", - "description": null, - "name": null - }, - { - "title": "param", - "description": "JSONPath referencing a point in `state`.", - "type": { - "type": "NameExpression", - "name": "String" - }, - "name": "path" - }, - { - "title": "returns", - "description": null, - "type": { - "type": "TypeApplication", - "expression": { - "type": "NameExpression", - "name": "Array" - }, - "applications": [ - { - "type": "UnionType", - "elements": [ - { - "type": "NameExpression", - "name": "String" - }, - { - "type": "NameExpression", - "name": "Object" - } - ] - } - ] - } - } - ] - }, - "valid": true - }, { "name": "dataPath", "params": [ @@ -481,59 +334,6 @@ }, "valid": true }, - { - "name": "each", - "params": [ - "dataSource", - "operation" - ], - "docs": { - "description": "Scopes an array of data based on a JSONPath.\nUseful when the source data has `n` items you would like to map to\nan operation.\nThe operation will receive a slice of the data based of each item\nof the JSONPath provided.\n\nIt also ensures the results of an operation make their way back into\nthe state's references.", - "tags": [ - { - "title": "public", - "description": null, - "type": null - }, - { - "title": "example", - "description": "each(\"$.[*]\",\n create(\"SObject\",\n field(\"FirstName\", sourceValue(\"$.firstName\"))\n )\n)" - }, - { - "title": "function", - "description": null, - "name": null - }, - { - "title": "param", - "description": "JSONPath referencing a point in `state`.", - "type": { - "type": "NameExpression", - "name": "DataSource" - }, - "name": "dataSource" - }, - { - "title": "param", - "description": "The operation needed to be repeated.", - "type": { - "type": "NameExpression", - "name": "Operation" - }, - "name": "operation" - }, - { - "title": "returns", - "description": null, - "type": { - "type": "NameExpression", - "name": "Operation" - } - } - ] - }, - "valid": true - }, { "name": "combine", "params": [ diff --git a/packages/common/src/Adaptor.js b/packages/common/src/Adaptor.js index 7e3105f64..7b08c84e3 100644 --- a/packages/common/src/Adaptor.js +++ b/packages/common/src/Adaptor.js @@ -13,8 +13,11 @@ export * as beta from './beta'; export * as http from './http.deprecated'; export * as dateFns from './dateFns'; +export { each, fn, sourceValue } from './operations' + const schemaCache = {}; + /** * Execute a sequence of operations. * Main outer API for executing expressions. @@ -38,34 +41,6 @@ export function execute(...operations) { }; } -/** - * alias for "fn()" - * @function - * @param {Function} func is the function - * @returns {Operation} - */ -export function alterState(func) { - return fn(func); -} - -/** - * Creates a custom step (or operation) for more flexible job writing. - * @public - * @example - * fn(state => { - * // do some things to state - * return state; - * }); - * @function - * @param {Function} func is the function - * @returns {Operation} - */ -export function fn(func) { - return state => { - return func(state); - }; -} - /** * Picks out a single value from a JSON object. * If a JSONPath returns more than one value for the reference, the first @@ -82,39 +57,8 @@ export function jsonValue(obj, path) { return JSONPath({ path, json: obj })[0]; } -/** - * Picks out a single value from source data. - * If a JSONPath returns more than one value for the reference, the first - * item will be returned. - * @public - * @example - * sourceValue('$.key') - * @function - * @param {String} path - JSONPath referencing a point in `state`. - * @returns {Operation} - */ -export function sourceValue(path) { - return state => { - return JSONPath({ path, json: state })[0]; - }; -} -/** - * Picks out a value from source data. - * Will return whatever JSONPath returns, which will always be an array. - * If you need a single value use `sourceValue` instead. - * @public - * @example - * source('$.key') - * @function - * @param {String} path - JSONPath referencing a point in `state`. - * @returns {Array.} - */ -export function source(path) { - return state => { - return JSONPath({ path, json: state }); - }; -} + /** * Ensures a path points at the data. @@ -230,7 +174,7 @@ export const map = curry(function (path, operation, state) { export function asData(data, state) { switch (typeof data) { case 'string': - return source(data)(state); + return JSONPath({ path: data, json: state }); case 'object': return data; case 'function': @@ -238,44 +182,7 @@ export function asData(data, state) { } } -/** - * Scopes an array of data based on a JSONPath. - * Useful when the source data has `n` items you would like to map to - * an operation. - * The operation will receive a slice of the data based of each item - * of the JSONPath provided. - * - * It also ensures the results of an operation make their way back into - * the state's references. - * @public - * @example - * each("$.[*]", - * create("SObject", - * field("FirstName", sourceValue("$.firstName")) - * ) - * ) - * @function - * @param {DataSource} dataSource - JSONPath referencing a point in `state`. - * @param {Operation} operation - The operation needed to be repeated. - * @returns {Operation} - */ -export function each(dataSource, operation) { - if (!dataSource) { - throw new TypeError('dataSource argument for each operation is invalid.'); - } - return state => { - return asData(dataSource, state).reduce((state, data, index) => { - if (state.then) { - return state.then(state => { - return operation({ ...state, data, index }); - }); - } else { - return operation({ ...state, data, index }); - } - }, state); - }; -} /** * Combines two operations into one diff --git a/packages/common/src/operations.js b/packages/common/src/operations.js new file mode 100644 index 000000000..bee3dcb48 --- /dev/null +++ b/packages/common/src/operations.js @@ -0,0 +1,101 @@ +// This file contains all the operations exported by the common adaptor +// ie, everything you can call from job code +import { JSONPath } from 'jsonpath-plus'; + +import { asData } from './Adaptor'; +import { operation } from './util'; + +/** + * Creates a custom step (or operation) for more flexible job writing. + * @public + * @example + * fn(state => { + * // do some things to state + * return state; + * }); + * @function + * @param {Function} func is the function + * @returns {Operation} + */ + +export const fn = operation((state, func) => { + return func(state); +}) + +/** + * alias for "fn()" + * @function + * @param {Function} func is the function + * @returns {Operation} + */ + +export const alterState = fn; + +// NOTE: docs shouldn't change, but I'd like to check that typings still work +// May need a little finessing +/** + * Picks out a single value from source data. + * If a JSONPath returns more than one value for the reference, the first + * item will be returned. + * @public + * @example + * sourceValue('$.key') + * @function + * @param {String} path - JSONPath referencing a point in `state`. + * @returns {Operation} + */ +export const sourceValue = operation((state, path) => { + return JSONPath({ path, json: state })[0]; +}) + +/** + * Picks out a value from source data. + * Will return whatever JSONPath returns, which will always be an array. + * If you need a single value use `sourceValue` instead. + * @public + * @example + * source('$.key') + * @function + * @param {String} path - JSONPath referencing a point in `state`. + * @returns {Array.} + */ +export const source = operation((state, path) => { + return JSONPath({ path, json: state }); +}) + +/** + * Scopes an array of data based on a JSONPath. + * Useful when the source data has `n` items you would like to map to + * an operation. + * The operation will receive a slice of the data based of each item + * of the JSONPath provided. + * + * It also ensures the results of an operation make their way back into + * the state's references. + * @public + * @example + * each("$.[*]", + * create("SObject", + * field("FirstName", sourceValue("$.firstName")) + * ) + * ) + * @function + * @param {DataSource} dataSource - JSONPath referencing a point in `state`. + * @param {Operation} operation - The operation needed to be repeated. + * @returns {Operation} + */ +export const each = operation((state, dataSource, fn) => { + if (!dataSource) { + throw new TypeError('dataSource argument for each fn is invalid.'); + } + + return asData(dataSource, state).reduce((state, data, index) => { + if (state.then) { + return state.then(state => { + return fn({ ...state, data, index }); + }); + } else { + return fn({ ...state, data, index }); + } + }, state); +}) \ No newline at end of file diff --git a/packages/common/src/util/index.js b/packages/common/src/util/index.js index a97522682..8a299fd21 100644 --- a/packages/common/src/util/index.js +++ b/packages/common/src/util/index.js @@ -1,2 +1,16 @@ export * from './http'; export * from './references'; + +// operation creates a new operation factory +// It accepts a function which takes user arguments AND state +// It returns a function which acccepts user args +// Which in turn returns a function which accepts state +// and then triggers the original function +// It's complex BUT the end user code should be way way simpler +export const operation = (fn) => { + return (...args) => { + return (state) => { + return fn(state, ...args) + } + } +} \ No newline at end of file diff --git a/packages/common/test/each.test.js b/packages/common/test/each.test.js index 37e448080..f2f009087 100644 --- a/packages/common/test/each.test.js +++ b/packages/common/test/each.test.js @@ -1,6 +1,7 @@ import { expect } from 'chai'; import testData from './fixtures/data.json' assert { type: 'json' }; -import { each, beta } from '../src/Adaptor'; +import { beta } from '../src/Adaptor'; +import { each } from '../src/operations'; function shouldBehaveLikeEach(each) { let state, operation; diff --git a/packages/common/test/index.test.js b/packages/common/test/index.test.js index 6938e75d6..a7f8d20be 100644 --- a/packages/common/test/index.test.js +++ b/packages/common/test/index.test.js @@ -9,7 +9,6 @@ import { combine, dataPath, dataValue, - each, execute, expandReferences, field, @@ -23,17 +22,53 @@ import { parseCsv, referencePath, scrubEmojis, - source, - sourceValue, splitKeys, toArray, validate, + + operation, } from '../src/Adaptor'; +import { + sourceValue, + each, + source, +} from '../src/operations'; const mockAgent = new MockAgent(); setGlobalDispatcher(mockAgent); const mockPool = mockAgent.get('https://localhost:1'); +// TODO move to the bottom or another file +describe('operation', () => { + it('should return an operation factory that works', async () => { + // declare an operation called fetch + const fetch = operation(async (state, url) => { + // do a simple request with undici and write it to state + const response = await request(url) + state.data = await response.body.text() + return state; + }) + + mockPool + .intercept({ + method: 'GET', + path: '/a', + }) + .reply(200, 'ok'); + + const state = {}; + + // run it + // Note that there's no way to mock the operation yet + // Answers are a) provide a dymanic mock callback + // b) provide an inner function which accepts a client + await fetch('https://localhost:1/a')(state) + + // check the state + expect(state.data).to.eql('ok') + }) +}) + describe('execute', () => { it('executes each operation in sequence', done => { let state = {}; diff --git a/packages/msgraph/examples/get-drive.js b/packages/msgraph/examples/get-drive.js new file mode 100644 index 000000000..8a64f0c83 --- /dev/null +++ b/packages/msgraph/examples/get-drive.js @@ -0,0 +1,8 @@ +/** + * A simple test job which gets a drive + */ +getDrive( + (state) => ({ id: state.id }), // get the drive id from state + "default", // drive name + (state) => ({ ...state, savedDrives: state.drives }) // alias savedDrives onto state (it gets removed by the adaptor) +) diff --git a/packages/msgraph/package.json b/packages/msgraph/package.json index b2bda9c3c..838cc3579 100644 --- a/packages/msgraph/package.json +++ b/packages/msgraph/package.json @@ -12,8 +12,8 @@ }, "scripts": { "build": "pnpm clean && build-adaptor msgraph", - "test": "mocha --experimental-specifier-resolution=node --no-warnings", - "test:watch": "mocha -w --experimental-specifier-resolution=node --no-warnings", + "test": "mocha --experimental-specifier-resolution=node --experimental-vm-modules --no-warnings", + "test:watch": "mocha -w --experimental-specifier-resolution=node --experimental-vm-modules --no-warnings", "clean": "rimraf dist types docs", "pack": "pnpm pack --pack-destination ../../dist", "lint": "eslint src" @@ -34,6 +34,7 @@ "devDependencies": { "@openfn/buildtools": "workspace:^1.0.2", "@openfn/simple-ast": "0.4.1", + "@openfn/test-execute": "workspace:^", "assertion-error": "2.0.0", "chai": "4.3.6", "deep-eql": "4.1.1", diff --git a/packages/msgraph/src/Adaptor.js b/packages/msgraph/src/Adaptor.js deleted file mode 100644 index f346b27dd..000000000 --- a/packages/msgraph/src/Adaptor.js +++ /dev/null @@ -1,384 +0,0 @@ -import { execute as commonExecute } from '@openfn/language-common'; -import { expandReferences } from '@openfn/language-common/util'; - -import { - request, - setUrl, - handleResponse, - assertDrive, - assertResources, -} from './Utils'; - -/** - * Execute a sequence of operations. - * Wraps `language-common/execute` to make working with this API easier. - * @example - * execute( - * create('foo'), - * delete('bar') - * )(state) - * @private - * @param {Operations} operations - Operations to be performed. - * @returns {Operation} - */ -export function execute(...operations) { - const initialState = { - references: [], - data: null, - drives: {}, - }; - - const cleanup = finalState => { - if (finalState?.buffer) { - delete finalState.buffer; - } - if (finalState?.drives) { - delete finalState.drives; - } - - return finalState; - }; - - return state => { - return commonExecute(...operations)({ - ...initialState, - ...state, - }) - .then(cleanup) - .catch(error => { - cleanup(state); - throw error; - }); - }; -} - -/** - * Create some resource in msgraph - * @public - * @example - * create("applications", {"displayName": "My App"}) - * @function - * @param {string} resource - The type of entity that will be created - * @param {object} data - The data to create the new resource - * @param {function} callback - An optional callback function - * @returns {Operation} - */ -export function create(resource, data, callback) { - return state => { - const [resolvedResource, resolvedData] = expandReferences( - state, - resource, - data - ); - - const { accessToken, apiVersion } = state.configuration; - - const url = setUrl({ apiVersion, resolvedResource }); - - const options = { - accessToken, - body: JSON.stringify(resolvedData), - method: 'POST', - }; - - return request(url, options).then(response => - handleResponse(response, state, callback) - ); - }; -} - -/** - * Make a GET request to msgraph resource - * @public - * @example - * get('sites/root/lists') - * @function - * @param {string} path - Path to resource - * @param {object} query - Query, Headers and Authentication parameters - * @param {function} callback - (Optional) Callback function - * @returns {Operation} - */ -export function get(path, query, callback = false) { - return state => { - const { accessToken, apiVersion } = state.configuration; - const [resolvedPath, resolvedQuery] = expandReferences(state, path, query); - - const url = setUrl(resolvedPath, apiVersion); - - return request(url, { query: resolvedQuery, accessToken }).then(response => - handleResponse(response, state, callback) - ); - }; -} - -/** - * Get a Drive or SharePoint document library. The drive metadata will be written - * to state.drives, where it can be used by other adaptor functions. - * Pass { id } to get a drive by id or { id, owner } to get default drive for - * some parent resource, like a group - * @public - * @example Get a drive by ID - * getDrive({ id: "YXzpkoLwR06bxC8tNdg71m" }) - * @example Get the default drive for a site - * getDrive({ id: "openfn.sharepoint.com", owner: "sites" }) - * @param specifier {Object} - A definition of the drive to retrieve - * - id {string} - The ID of the resource or owner. - * - owner {string} - The type of drive owner (e.g. sites, groups). - * @param {string} name - The local name of the drive used to write to state.drives, ie, state.drives[name] - * @param {function} [callback = s => s] (Optional) Callback function - * @return {Operation} - */ -export function getDrive(specifier, name = 'default', callback = s => s) { - return state => { - const { accessToken, apiVersion } = state.configuration; - const [resolvedSpecifier, resolvedName] = expandReferences( - state, - specifier, - name - ); - - const { id, owner = 'drive' } = resolvedSpecifier; - - let resource; - if (owner === 'drive') { - resource = `drives/${id}`; - } else { - resource = `${owner}/${id}/drive`; - } - - const url = setUrl(resource, apiVersion); - - return request(url, { accessToken }).then(response => { - state.drives[resolvedName] = response; - return callback(state); - }); - }; -} - -/** - * Get the contents or metadata of a folder. - * @public - * @example Get a folder by ID - * getFolder('01LUM6XOCKDTZKQC7AVZF2VMHE2I3O6OY3') - * @example Get a folder for a named drive by id - * getFolder("01LUM6XOCKDTZKQC7AVZF2VMHE2I3O6OY3",{ driveName: "mydrive"}) - * @param {string} pathOrId - A path to a folder or folder id - * @param {object} options - (Optional) Query parameters - * @param {function} [callback = s => s] (Optional) Callback function - * @return {Operation} - */ -export function getFolder(pathOrId, options, callback = s => s) { - return async state => { - const defaultOptions = { - driveName: 'default', // Named drive in state.drives - metadata: false, // If false return folder files if true return folder metadata - // $filter: '', // Eg: "file/mimeType eq \'application/vnd.ms-excel\'" - }; - const { accessToken, apiVersion } = state.configuration; - const [resolvedPathOrId, resolvedOptions] = expandReferences( - state, - pathOrId, - options - ); - - const { driveName, metadata } = { ...defaultOptions, ...resolvedOptions }; - - assertDrive(state, driveName); - - const { id: driveId } = state.drives[driveName]; - - let resource; - - if (resolvedPathOrId.startsWith('/')) { - resource = `drives/${driveId}/root:/${encodeURIComponent( - resolvedPathOrId - )}`; - } else { - resource = `drives/${driveId}/items/${resolvedPathOrId}`; - } - - if (!metadata) { - resource += resolvedPathOrId.startsWith('/') ? ':/children' : '/children'; - } - - const url = setUrl(resource, apiVersion); - - return request(url, { accessToken }).then(response => - handleResponse(response, state, callback) - ); - }; -} - -/** - * Get file metadata or file content. - * @public - * @example Get a file by ID - * getFile('01LUM6XOGRONYNTZ26DBBJPTN5IFTQPBIW') - * @example Get a file for a named drive by id - * getFile("01LUM6XOGRONYNTZ26DBBJPTN5IFTQPBIW",{ driveName: "mydrive"}) - * @param {string} pathOrId - A path to a file or file id - * @param {object} options - (Optional) Query parameters - * @param {function} [callback = s => s] (Optional) Callback function - * @return {Operation} - */ -export function getFile(pathOrId, options, callback = s => s) { - const defaultOptions = { - driveName: 'default', // named drive in state.drives - metadata: false, // Returns file msgraph metadata - // $filter: '', // Eg: "file/mimeType eq \'application/vnd.ms-excel\'" - // select: '', // Eg: id,@microsoft.graph.downloadUrl - }; - return async state => { - const { accessToken, apiVersion } = state.configuration; - const [resolvedPathOrId, resolvedOptions] = expandReferences( - state, - pathOrId, - options - ); - - const { driveName, metadata } = { - ...defaultOptions, - ...resolvedOptions, - }; - - assertDrive(state, driveName); - - const { id: driveId } = state.drives[driveName]; - - let resource; - - if (resolvedPathOrId.startsWith('/')) { - resource = `drives/${driveId}/root:/${encodeURIComponent( - resolvedPathOrId - )}`; - } else { - resource = `drives/${driveId}/items/${resolvedPathOrId}`; - } - - if (!metadata) { - resource += resolvedPathOrId.startsWith('/') ? ':/content' : '/content'; - } - - const url = setUrl(resource, apiVersion); - - const response = await request(url, { - accessToken, - parseAs: metadata ? 'json' : 'text', - }); - - return handleResponse(response, state, callback); - }; -} - -const defaultResource = { - contentType: 'application/octet-stream', - driveId: '', - folderId: '', - fileName: 'sheet.xls', - onConflict: 'replace', -}; - -/** - * Upload a file to a drive - * @public - * @example - * Upload Excel file to a drive using `driveId` and `parantItemId` - * uploadFile( - * state => ({ - * driveId: state.driveId, - * folderId: state.folderId, - * fileName: `Tracker.xlsx`, - * }), - * state => state.buffer - * ); - * @example - * Upload Excel file to a SharePoint drive using `siteId` and `parantItemId` - * uploadFile( - * state => ({ - * siteId: state.siteId, - * folderId: state.folderId, - * fileName: `Report.xlsx`, - * }), - * state => state.buffer - * ); - * @function - * @param {Object} resource - Resource Object - * @param {String} [resource.driveId] - Drive Id - * @param {String} [resource.driveId] - Site Id - * @param {String} [resource.folderId] - Parent folder id - * @param {String} [resource.contentType] - Resource content-type - * @param {String} [resource.onConflict] - Specify conflict behavior if file with the same name exists. Can be "rename | fail | replace" - * @param {Object} data - A buffer containing the file. - * @param {Function} callback - Optional callback function - * @returns {Operation} - */ -export function uploadFile(resource, data, callback) { - return async state => { - const { accessToken, apiVersion } = state.configuration; - - const [resolvedResource, resolvedData] = expandReferences( - state, - resource, - data - ); - - const { contentType, driveId, siteId, folderId, onConflict, fileName } = { - ...defaultResource, - ...resolvedResource, - }; - - assertResources({ driveId, siteId, folderId }); - - const path = - (driveId && - `drives/${driveId}/items/${folderId}:/${fileName}:/createUploadSession`) || - (siteId && - `sites/${siteId}/drive/items/${folderId}:/${fileName}:/createUploadSession`); - - const uploadSession = await request(setUrl(path, apiVersion), { - method: 'POST', - accessToken, - headers: { - 'Content-Type': 'application/json', - }, - body: JSON.stringify({ - '@microsoft.graph.conflictBehavior': onConflict, - name: fileName, - }), - }); - - const uploadUrl = uploadSession.uploadUrl; - - console.log(`Uploading file...`); - - return request(uploadUrl, { - method: 'PUT', - accessToken, - headers: { - 'Content-Type': contentType, - 'Content-Length': `${resolvedData.length}`, - 'Content-Range': `bytes 0-${resolvedData.length - 1}/${ - resolvedData.length - }`, - }, - - body: resolvedData, - }).then(response => handleResponse(response, state, callback)); - }; -} - -export { request, sheetToBuffer } from './Utils'; - -export { - dataPath, - dataValue, - dateFns, - each, - field, - fields, - fn, - lastReferenceValue, - merge, - sourceValue, - parseCsv, -} from '@openfn/language-common'; diff --git a/packages/msgraph/src/impl.js b/packages/msgraph/src/impl.js new file mode 100644 index 000000000..9cbc6a94d --- /dev/null +++ b/packages/msgraph/src/impl.js @@ -0,0 +1,216 @@ +import { handleResponse, setUrl } from './utils'; +import { expandReferences } from '@openfn/language-common/util'; + +// Default the callback here in the impl, so that unit tests +// can optionall exclude it +const noop = s => s; + +// This is a totally normal js function +// It can be unit tested by passing a request function in +export const create = (state, request, resource, data, callback = noop) => { + const [resolvedResource, resolvedData] = expandReferences( + state, + resource, + data + ); + + const { accessToken, apiVersion } = state.configuration; + + const url = setUrl({ apiVersion, resolvedResource }); + + const options = { + accessToken, + body: JSON.stringify(resolvedData), + method: 'POST', + }; + + return request(url, options).then(response => + handleResponse(response, state, callback) + ); +} + +export const get = (state, request, path, query) => { + const { accessToken, apiVersion } = state.configuration; + const [resolvedPath, resolvedQuery] = expandReferences(state, path, query); + + const url = setUrl(resolvedPath, apiVersion); + + return request(url, { query: resolvedQuery, accessToken }).then(response => + handleResponse(response, state, callback) + ); +} + +export const getDrive = (state, request, specifier, name = 'default', callback = noop) => { + const { accessToken, apiVersion } = state.configuration; + const [resolvedSpecifier, resolvedName] = expandReferences( + state, + specifier, + name + ); + + const { id, owner = 'drive' } = resolvedSpecifier; + + let resource; + if (owner === 'drive') { + resource = `drives/${id}`; + } else { + resource = `${owner}/${id}/drive`; + } + + const url = setUrl(resource, apiVersion); + + return request(url, { accessToken }).then(response => { + state.drives[resolvedName] = response; + return callback(state); + }); +} + +export const getFolder = (state, request, pathOrId, options, callback = noop) => { + const defaultOptions = { + driveName: 'default', // Named drive in state.drives + metadata: false, // If false return folder files if true return folder metadata + // $filter: '', // Eg: "file/mimeType eq \'application/vnd.ms-excel\'" + }; + const { accessToken, apiVersion } = state.configuration; + const [resolvedPathOrId, resolvedOptions] = expandReferences( + state, + pathOrId, + options + ); + + const { driveName, metadata } = { ...defaultOptions, ...resolvedOptions }; + + assertDrive(state, driveName); + + const { id: driveId } = state.drives[driveName]; + + let resource; + + if (resolvedPathOrId.startsWith('/')) { + resource = `drives/${driveId}/root:/${encodeURIComponent( + resolvedPathOrId + )}`; + } else { + resource = `drives/${driveId}/items/${resolvedPathOrId}`; + } + + if (!metadata) { + resource += resolvedPathOrId.startsWith('/') ? ':/children' : '/children'; + } + + const url = setUrl(resource, apiVersion); + + return request(url, { accessToken }).then(response => + handleResponse(response, state, callback) + ); +} + +export const getFile = async (state, request, pathOrId, options, callback) => { + const defaultOptions = { + driveName: 'default', // named drive in state.drives + metadata: false, // Returns file msgraph metadata + // $filter: '', // Eg: "file/mimeType eq \'application/vnd.ms-excel\'" + // select: '', // Eg: id,@microsoft.graph.downloadUrl + }; + + const { accessToken, apiVersion } = state.configuration; + const [resolvedPathOrId, resolvedOptions] = expandReferences( + state, + pathOrId, + options + ); + + const { driveName, metadata } = { + ...defaultOptions, + ...resolvedOptions, + }; + + assertDrive(state, driveName); + + const { id: driveId } = state.drives[driveName]; + + let resource; + + if (resolvedPathOrId.startsWith('/')) { + resource = `drives/${driveId}/root:/${encodeURIComponent( + resolvedPathOrId + )}`; + } else { + resource = `drives/${driveId}/items/${resolvedPathOrId}`; + } + + if (!metadata) { + resource += resolvedPathOrId.startsWith('/') ? ':/content' : '/content'; + } + + const url = setUrl(resource, apiVersion); + + const response = await request(url, { + accessToken, + parseAs: metadata ? 'json' : 'text', + }); + + return handleResponse(response, state, callback); +} + +export const uploadFile = async (state, request, resource, data, callback) => { + const defaultResource = { + contentType: 'application/octet-stream', + driveId: '', + folderId: '', + fileName: 'sheet.xls', + onConflict: 'replace', + }; + + const { accessToken, apiVersion } = state.configuration; + + const [resolvedResource, resolvedData] = expandReferences( + state, + resource, + data + ); + + const { contentType, driveId, siteId, folderId, onConflict, fileName } = { + ...defaultResource, + ...resolvedResource, + }; + + assertResources({ driveId, siteId, folderId }); + + const path = + (driveId && + `drives/${driveId}/items/${folderId}:/${fileName}:/createUploadSession`) || + (siteId && + `sites/${siteId}/drive/items/${folderId}:/${fileName}:/createUploadSession`); + + const uploadSession = await request(setUrl(path, apiVersion), { + method: 'POST', + accessToken, + headers: { + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + '@microsoft.graph.conflictBehavior': onConflict, + name: fileName, + }), + }); + + const uploadUrl = uploadSession.uploadUrl; + + console.log(`Uploading file...`); + + const response = await request(uploadUrl, { + method: 'PUT', + accessToken, + headers: { + 'Content-Type': contentType, + 'Content-Length': `${resolvedData.length}`, + 'Content-Range': `bytes 0-${resolvedData.length - 1}/${ + resolvedData.length + }`, + }, + body: resolvedData, + }) + + return handleResponse(response, state, callback); +} \ No newline at end of file diff --git a/packages/msgraph/src/index.js b/packages/msgraph/src/index.js index a013b3648..4d2c22fd4 100644 --- a/packages/msgraph/src/index.js +++ b/packages/msgraph/src/index.js @@ -1,4 +1,2 @@ -import * as Adaptor from './Adaptor'; +import * as Adaptor from './operations'; export default Adaptor; - -export * from './Adaptor'; \ No newline at end of file diff --git a/packages/msgraph/test/fixtures.js b/packages/msgraph/src/mock/fixtures.js similarity index 100% rename from packages/msgraph/test/fixtures.js rename to packages/msgraph/src/mock/fixtures.js diff --git a/packages/msgraph/src/mock/mock.js b/packages/msgraph/src/mock/mock.js new file mode 100644 index 000000000..d5b6cb4d1 --- /dev/null +++ b/packages/msgraph/src/mock/mock.js @@ -0,0 +1,70 @@ +import { setGlobalDispatcher, MockAgent } from 'undici'; +import { fixtures } from './fixtures'; + +/** + * This provides a mock interface to the msgraph client + * + * each function should be overridable through a common setMock API + * + * We need to document this in a way that the playground can use + */ + +let mockAgent; +let mockPool; + +// To make custom overrides easier, export some stanadard patterns +export const patterns = { + // TODO the regex might be a bit hard here, as we + // need to distinguish the get drive patterns + drives: /\/drives\// +} + +// name: { pattern, data, options } +const defaultRoutes = { + 'drives': { pattern: patterns.drives, data: fixtures.driveResponse } +} + +export const enable = (state, routes = {}) => { + // TODO if an agent already exists, should we destroy it? + + // set the global dispacher on undici + mockAgent = new MockAgent(); + + // Don't let the agentcall out to the real internet + mockAgent.disableNetConnect(); + + setGlobalDispatcher(mockAgent); + + mockPool = mockAgent.get('https://graph.microsoft.com'); + + const mockRoutes = { + ...defaultRoutes, + ...routes + } + + // Set up all the mock routes + for (const name in mockRoutes) { + const { pattern, data, options } = mockRoutes[name]; + mockRoute(pattern, data, options) + } +} + +// API to enable a particular path to be mocked +export const mockRoute = (path, data, options = {}) => { + const jsonResponse = { + headers: { + 'Content-Type': 'application/json', + }, + }; + + const { method = 'GET', headers, once = false } = options + const scope = mockPool.intercept({ + path, + method, + headers, + }).reply(200, JSON.stringify(data),jsonResponse) + + if (!once) { + scope.persist() + } +} \ No newline at end of file diff --git a/packages/msgraph/src/operations.js b/packages/msgraph/src/operations.js new file mode 100644 index 000000000..8794e3812 --- /dev/null +++ b/packages/msgraph/src/operations.js @@ -0,0 +1,185 @@ +import { execute as commonExecute } from '@openfn/language-common' +import { operation } from '@openfn/language-common/util' + +import * as impl from './impl'; +import { request } from './utils'; +import { enable } from './mock/mock'; + +// The runtime itself will call this to flick the whole thing into mock mode +export const enableMock = (state, routes) => { + return enable(state, routes); +} +/** + * Execute a sequence of operations. + * Wraps `language-common/execute` to make working with this API easier. + */ +export function execute(...operations) { + const initialState = { + references: [], + data: null, + drives: {}, + }; + + const cleanup = finalState => { + if (finalState?.buffer) { + delete finalState.buffer; + } + if (finalState?.drives) { + delete finalState.drives; + } + + return finalState; + }; + + return state => { + return commonExecute(...operations)({ + ...initialState, + ...state, + }) + .then(cleanup) + .catch(error => { + cleanup(state); + throw error; + }); + }; +} + +/** + * Create some resource in msgraph. + * @public + * @example + * create("applications", {"displayName": "My App"}) + * @function + * @param {string} resource - The type of entity that will be created + * @param {object} data - The data to create the new resource + * @param {function} callback - An optional callback function + * @returns {Operation} + */ +export const create = operation((state, resource, data, callback) => { + return impl.create(state, request, resource, data, callback) +}) + +/** + * Make a GET request to msgraph resource + * @public + * @example + * get('sites/root/lists') + * @function + * @param {string} path - Path to resource + * @param {object} query - Query, Headers and Authentication parameters + * @param {function} callback - (Optional) Callback function + * @returns {Operation} + */ +export const get = operation((state, path, query, callback) => { + return impl.get(state, request, path, query, callback) +}) + + +/** + * Get a Drive or SharePoint document library. The drive metadata will be written + * to state.drives, where it can be used by other adaptor functions. + * Pass { id } to get a drive by id or { id, owner } to get default drive for + * some parent resource, like a group. + * @public + * @example Get a drive by ID + * getDrive({ id: "YXzpkoLwR06bxC8tNdg71m" }) + * @example Get the default drive for a site + * getDrive({ id: "openfn.sharepoint.com", owner: "sites" }) + * @param specifier {Object} - A definition of the drive to retrieve + * - id {string} - The ID{ defa} of the resource or owner. + * - owner {string} - The type of drive owner (e.g. sites, groups). + * @param {string} name - The local name of the drive used to write to state.drives, ie, state.drives[name] + * @param {function} [callback = s => s] (Optional) Callback function + * @return {Operation} + */ +export const getDrive = operation((state, specifier, name, callback) => { + return impl.getDrive(state, request, specifier, name, callback) +}) + +/** + * Get the contents or metadata of a folder. + * @public + * @example Get a folder by ID + * getFolder('01LUM6XOCKDTZKQC7AVZF2VMHE2I3O6OY3') + * @example Get a folder for a named drive by id + * getFolder("01LUM6XOCKDTZKQC7AVZF2VMHE2I3O6OY3",{ driveName: "mydrive"}) + * @param {string} pathOrId - A path to a folder or folder id + * @param {object} options - (Optional) Query parameters + * @param {function} [callback = s => s] (Optional) Callback function + * @return {Operation} + */ +export const getFolder = operation((state, pathOrId, options, callback) => { + return impl.getFolder(state, request, pathOrId, options, callback) +}); + + +/** + * Get file metadata or file content. + * @public + * @example Get a file by ID + * getFile('01LUM6XOGRONYNTZ26DBBJPTN5IFTQPBIW') + * @example Get a file for a named drive by id + * getFile("01LUM6XOGRONYNTZ26DBBJPTN5IFTQPBIW",{ driveName: "mydrive"}) + * @param {string} pathOrId - A path to a file or file id + * @param {object} options - (Optional) Query parameters + * @param {function} [callback = s => s] (Optional) Callback function + * @return {Operation} + */ +export const getFile = operation((state, pathOrId, options, callback) => { + return impl.getFile(state, request, pathOrId, options, callback) +}); + +/** + * Upload a file to a drive. + * @public + * @example + * Upload Excel file to a drive using `driveId` and `parantItemId` + * uploadFile( + * state => ({ + * driveId: state.driveId, + * folderId: state.folderId, + * fileName: `Tracker.xlsx`, + * }), + * state => state.buffer + * ); + * @example + * Upload Excel file to a SharePoint drive using `siteId` and `parantItemId` + * uploadFile( + * state => ({ + * siteId: state.siteId, + * folderId: state.folderId, + * fileName: `Report.xlsx`, + * }), + * state => state.buffer + * ); + * @function + * @param {Object} resource - Resource Object + * @param {String} [resource.driveId] - Drive Id + * @param {String} [resource.driveId] - Site Id + * @param {String} [resource.folderId] - Parent folder id + * @param {String} [resource.contentType] - Resource content-type + * @param {String} [resource.onConflict] - Specify conflict behavior if file with the same name exists. Can be "rename | fail | replace" + * @param {Object} data - A buffer containing the file. + * @param {Function} callback - Optional callback function + * @returns {Operation} + */ +export const uploadFile = operation((state, resource, data, callback) => { + return impl.uploadFile(state, request, pathOrId, options, callback) +}); + +export { request, sheetToBuffer } from './utils'; + +export { + dataPath, + dataValue, + dateFns, + each, + field, + fields, + fn, + lastReferenceValue, + merge, + sourceValue, + parseCsv, +} from '@openfn/language-common'; + diff --git a/packages/msgraph/src/Utils.js b/packages/msgraph/src/utils.js similarity index 95% rename from packages/msgraph/src/Utils.js rename to packages/msgraph/src/utils.js index f741b27b7..0efe53173 100644 --- a/packages/msgraph/src/Utils.js +++ b/packages/msgraph/src/utils.js @@ -1,4 +1,3 @@ -import xlsx from 'xlsx'; import { fetch } from 'undici'; import { Readable, Writable } from 'node:stream'; import { composeNextState, asData } from '@openfn/language-common'; @@ -50,22 +49,19 @@ const isStream = value => { return false; }; -export function handleResponse(response, state, callback) { - let nextState; +export function handleResponse(response, state, callback = (s) => s) { // Don't compose state if response is a stream if (isStream(response)) { - nextState = { + return callback({ ...state, data: response, - }; - } else { - nextState = { - ...composeNextState(state, response), - response, - }; + }); } - if (callback) return callback(nextState); - return nextState; + + callback({ + ...composeNextState(state, response), + response, + }); } export function handleResponseError(response, data, method) { diff --git a/packages/msgraph/test/Adaptor.test.js b/packages/msgraph/test/Adaptor.deprecated similarity index 100% rename from packages/msgraph/test/Adaptor.test.js rename to packages/msgraph/test/Adaptor.deprecated diff --git a/packages/msgraph/test/examples.test.js b/packages/msgraph/test/examples.test.js new file mode 100644 index 000000000..3c7300486 --- /dev/null +++ b/packages/msgraph/test/examples.test.js @@ -0,0 +1,44 @@ +import { expect } from 'chai'; +import execute from '@openfn/test-execute'; + +import Adaptor from '../src/index'; +import { fixtures } from '../src/mock/fixtures'; + +const configuration = { + // The mock could test for these things and throw appropriately if I wanted + accessToken: 'a', + apiVersion: '1', +}; + +// TODO move to tools +const loadExample = (name) => { + // TODO we cheat for now, no-one is watching + return 'getDrive((state) => ({ id: state.id }), "default", (state) => ({ ...state, savedDrives: state.drives }))' +} + +// Configure the adaptor to use default mocks for these tests +Adaptor.enableMock({ configuration }); + +describe('examples', () => { + it('get-drive', async () => { + // Load our example code + const source = loadExample('get-drive') + + // Set up some input state + const state = { + id: 'xxx', + configuration + }; + + // Compile and run the job against this adaptor + const finalState = await execute(source, Adaptor, state) + + // Make some final assertion against state + + // Remember that state.drives is actually removed... + expect(finalState.drives).to.be.undefined + + // So our job re-writes it + expect(finalState.savedDrives).to.eql({ default: fixtures.driveResponse }); + }) +}); \ No newline at end of file diff --git a/packages/msgraph/test/impl.test.js b/packages/msgraph/test/impl.test.js new file mode 100644 index 000000000..66bfac699 --- /dev/null +++ b/packages/msgraph/test/impl.test.js @@ -0,0 +1,24 @@ +import { expect } from 'chai'; +import * as impl from '../src/impl'; +import { fixtures } from '../src/mock/fixtures'; + +describe('getDrive', () => { + // look how simple this unit test is now + it('should get a drive by id and set it to state', async () => { + // no config needed, we take that for granted here + const state = { + configuration: {}, + drives: {} // TODO I shouldn't need to set this though + }; + + // Mock out request to just return the drive response data + // TODO if we want, we cah test crentials here + const request = async () => fixtures.driveResponse; + + // We can literally just test the result of get drive + const result = await impl.getDrive(state, request, { id: 'b!YXzpkoLwR06bxC8tNdg71m_' }) + + // TODO not sure why this is wrapped in default... + expect(result.drives).to.eql({ default: fixtures.driveResponse }); + }); +}); \ No newline at end of file diff --git a/packages/msgraph/test/mock.test.js b/packages/msgraph/test/mock.test.js new file mode 100644 index 000000000..b485f4dfe --- /dev/null +++ b/packages/msgraph/test/mock.test.js @@ -0,0 +1,54 @@ +/** + * these tests check default and custom mock values + * they do not exercise execute or anything + */ +import { expect } from 'chai'; + +import { fixtures } from '../src/mock/fixtures'; +import { patterns } from '../src/mock/mock'; + +// not sure if this is the right import +// I guess it's fine +import Adaptor from '../src/index.js' + +const defaultState = { configuration: {resource: 'x'}, drives: {}}; + + +// Test all the default mock behaviours +describe('default values', () => { + before(() => { + Adaptor.enableMock(defaultState); + }); + + // These can all use the default mock + it('getDrive', async () => { + const state = { ...defaultState }; + + const result = await Adaptor.getDrive({ id: 'b!YXzpkoLwR06bxC8tNdg71m_' })(state) + expect(result.drives).to.eql({ default: fixtures.driveResponse }) + }) + + it('getDrive basically ignores the drive id argument', async () => { + const state = { ...defaultState }; + + const result = await Adaptor.getDrive({ id: 'abcdefg' })(state) + expect(result.drives).to.eql({ default: fixtures.driveResponse }) + }) +}) + +describe('custom values', () => { + it('getDrive', async () => { + Adaptor.enableMock(defaultState, { + 'drives': { + pattern: patterns.drives, + data: { x: 22 }, + options: { once: true } + } + }); + + const state = { ...defaultState }; + + const result = await Adaptor.getDrive({ id: 'b!YXzpkoLwR06bxC8tNdg71m_' })(state) + expect(result.drives).to.eql({ default: { x: 22 } }) + }) +}) \ No newline at end of file diff --git a/packages/msgraph/test/mockAgent.js b/packages/msgraph/test/mockAgent.deprecated similarity index 100% rename from packages/msgraph/test/mockAgent.js rename to packages/msgraph/test/mockAgent.deprecated diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1f4b1886e..6f4ac8df7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -1120,6 +1120,9 @@ importers: '@openfn/simple-ast': specifier: 0.4.1 version: 0.4.1 + '@openfn/test-execute': + specifier: workspace:^ + version: link:../../tools/execute assertion-error: specifier: 2.0.0 version: 2.0.0 @@ -2082,6 +2085,15 @@ importers: specifier: 17.6.0 version: 17.6.0 + tools/execute: + dependencies: + '@openfn/compiler': + specifier: ^0.0.32 + version: 0.0.32 + '@openfn/runtime': + specifier: ^1.0.0 + version: 1.1.1 + tools/import-tests: dependencies: '@openfn/language-common': @@ -3168,6 +3180,74 @@ packages: resolution: {integrity: sha512-ZnQMnLV4e7hDlUvw8H+U8ASL02SS2Gn6+9Ac3wGGLIe7+je2AeAOxPY+izIPJDfFDb7eDjev0Us8MO1iFRN8hA==} dev: true + /@inquirer/confirm@0.0.28-alpha.0: + resolution: {integrity: sha512-ZpQQMRt0yign/M2F/PRv+RwnjRhLZz2OIU3ohhDS0H6LoY2/W6xiPJpRJYxb15KN8n/QRql0yXzPg0ugeHCwKg==} + dependencies: + '@inquirer/core': 0.0.30-alpha.0 + '@inquirer/input': 0.0.28-alpha.0 + chalk: 5.2.0 + dev: false + + /@inquirer/confirm@2.0.6: + resolution: {integrity: sha512-1lPtPRq/1so8wmND43QTIn+hg5WIPpy2u3b8G2MveQ6B1Y2pm6/2Q5DEEt2ndi0kfidjPwQEjfGMlUNcXzQQVw==} + engines: {node: '>=14.18.0'} + dependencies: + '@inquirer/core': 3.1.2 + '@inquirer/type': 1.2.0 + chalk: 4.1.2 + dev: false + + /@inquirer/core@0.0.30-alpha.0: + resolution: {integrity: sha512-bLDyC8LA+Aqy0/uAo9pm53qRFBvFexdo5Z1MTXbMAniNB8SeC6HtQnd4TRCJQVYbpR4C//xVtqB+jOD3oLSaCw==} + dependencies: + '@inquirer/type': 0.0.4-alpha.0 + ansi-escapes: 6.2.0 + chalk: 5.2.0 + cli-spinners: 2.9.2 + cli-width: 4.1.0 + lodash: 4.17.21 + mute-stream: 0.0.8 + run-async: 2.4.1 + string-width: 5.1.2 + strip-ansi: 7.1.0 + wrap-ansi: 8.1.0 + dev: false + + /@inquirer/core@3.1.2: + resolution: {integrity: sha512-lR2GaqBkp42Ew9BOAOqf2pSp+ymVES1qN8OC90WWh45yeoYLl0Ty1GyCxmkKqBJtq/+Ea1MF12AdFcZcpRNFsw==} + engines: {node: '>=14.18.0'} + dependencies: + '@inquirer/type': 1.2.0 + '@types/mute-stream': 0.0.1 + '@types/node': 20.5.0 + '@types/wrap-ansi': 3.0.0 + ansi-escapes: 4.3.2 + chalk: 4.1.2 + cli-spinners: 2.9.2 + cli-width: 4.1.0 + figures: 3.2.0 + mute-stream: 1.0.0 + run-async: 3.0.0 + strip-ansi: 6.0.1 + wrap-ansi: 6.2.0 + dev: false + + /@inquirer/input@0.0.28-alpha.0: + resolution: {integrity: sha512-jbMvmAY7irZFQco9i1+H+S/yMozOEvBp+gYgk1zhP+1J/4dBywCi3E/s0U6eq/U3CUm5HaxkRphl9d9OP13Lpg==} + dependencies: + '@inquirer/core': 0.0.30-alpha.0 + chalk: 5.2.0 + dev: false + + /@inquirer/type@0.0.4-alpha.0: + resolution: {integrity: sha512-D+6Z4o89zClJkfM6tMaASjiS29YzAMi18/ZgG1nxUhMLjldSnnRUw6EceIqv4fZp5PL2O6MyZkcV9c4GgREdKg==} + dev: false + + /@inquirer/type@1.2.0: + resolution: {integrity: sha512-/vvkUkYhrjbm+RolU7V1aUFDydZVKNKqKHR5TsE+j5DXgXFwrsOPcoGUJ02K0O7q7O53CU2DOTMYCHeGZ25WHA==} + engines: {node: '>=18'} + dev: false + /@jridgewell/gen-mapping@0.1.1: resolution: {integrity: sha512-sQXCasFk+U8lWYEe66WxRDOE9PjVz4vSM51fTu3Hw+ClTpUSQb718772vH3pyS5pShp6lvQM7SxgIDXXXmOX7w==} engines: {node: '>=6.0.0'} @@ -3280,6 +3360,63 @@ packages: '@nodelib/fs.scandir': 2.1.5 fastq: 1.13.0 + /@openfn/compiler@0.0.32: + resolution: {integrity: sha512-wrMORSFLSHNzu+Eh7Yj/Or17rMaBDovW5k/ilRmtUtc0Xw8ducR1vqXN1eL4OGXBmenjCLVtXaaGA0M/MwkXVw==} + engines: {node: '>=16', pnpm: '>=7'} + dependencies: + '@openfn/describe-package': 0.0.16 + '@openfn/logger': 0.0.13 + acorn: 8.8.1 + ast-types: 0.14.2 + recast: 0.21.5 + yargs: 17.7.2 + transitivePeerDependencies: + - encoding + - supports-color + dev: false + + /@openfn/describe-package@0.0.16: + resolution: {integrity: sha512-q7MILQ/f0V3tRBMC2DxIrYp1WXvMy0SUHs3ujPv1+zNDAPCPiatnsc3UO/q8S3BA6xt6HCss0yhFdv8p7YBl+Q==} + engines: {node: '>=16', pnpm: '>=7'} + dependencies: + '@typescript/vfs': 1.5.0 + cross-fetch: 3.1.8 + node-localstorage: 2.2.1 + typescript: 4.8.4 + url-join: 5.0.0 + transitivePeerDependencies: + - encoding + - supports-color + dev: false + + /@openfn/logger@0.0.13: + resolution: {integrity: sha512-5xx2D6CJBKohhP8AENItOT8nDBUvlkeSeMm6xfxcaXcjdhn9Ff2kN75dTKbs5KdgB4GNgMOl0zb3Ju4l2HXaIA==} + engines: {node: '>=16'} + dependencies: + '@inquirer/confirm': 0.0.28-alpha.0 + chalk: 5.2.0 + fast-safe-stringify: 2.1.1 + figures: 5.0.0 + dev: false + + /@openfn/logger@1.0.1: + resolution: {integrity: sha512-WWWs5pkdQDFwGQJInMwL0D2cN17j8vkuYsVkLnsoNzlgfK5GdsDfgmtA8i7ayLgHbupQ9dcvfMiZ/lXpOAiaSQ==} + engines: {node: '>=16'} + dependencies: + '@inquirer/confirm': 2.0.6 + chalk: 5.2.0 + fast-safe-stringify: 2.1.1 + figures: 5.0.0 + dev: false + + /@openfn/runtime@1.1.1: + resolution: {integrity: sha512-K8IMd0xsjC4LKXOtTEstDDt5YSzgeQHpCRodwpiMsXE4j/yvRG0iqFUt2tdLY8IRPkUFA1C6+TXGufdosR/O2A==} + dependencies: + '@openfn/logger': 1.0.1 + fast-safe-stringify: 2.1.1 + semver: 7.6.0 + dev: false + /@openfn/simple-ast@0.3.2: resolution: {integrity: sha512-NIvZsKSBQmGjQwqv8uDFpsTQquHkpoBH09pg+SJsInoa4L8CEW1g+ZU2O9D+i4xYeNciYb1nsfJ9n9TjxYAvzg==} hasBin: true @@ -3481,6 +3618,12 @@ packages: resolution: {integrity: sha512-jhuKLIRrhvCPLqwPcx6INqmKeiA5EWrsCOPhrlFSrbrmU4ZMPjj5Ul/oLCMDO98XRUIwVm78xICz4EPCektzeQ==} dev: true + /@types/mute-stream@0.0.1: + resolution: {integrity: sha512-0yQLzYhCqGz7CQPE3iDmYjhb7KMBFOP+tBkyw+/Y2YyDI5wpS7itXXxneN1zSsUwWx3Ji6YiVYrhAnpQGS/vkw==} + dependencies: + '@types/node': 18.17.5 + dev: false + /@types/node-fetch@2.6.10: resolution: {integrity: sha512-PPpPK6F9ALFTn59Ka3BaL+qGuipRfxNE8qVgkp0bVixeiR2c2/L+IVOiBdu9JhhT22sWnQEp6YyHGI2b2+CMcA==} dependencies: @@ -3536,6 +3679,18 @@ packages: '@types/node': 18.17.5 dev: false + /@types/wrap-ansi@3.0.0: + resolution: {integrity: sha512-ltIpx+kM7g/MLRZfkbL7EsCEjfzCcScLpkg37eXEtx5kmrAKBkTJwd1GIAjDSL8wTpM6Hzn5YO4pSb91BEwu1g==} + dev: false + + /@typescript/vfs@1.5.0: + resolution: {integrity: sha512-AJS307bPgbsZZ9ggCT3wwpg3VbTKMFNHfaY/uF0ahSkYYrPF2dSSKDNIDIQAHm9qJqbLvCsSJH7yN4Vs/CsMMg==} + dependencies: + debug: 4.3.4(supports-color@8.1.1) + transitivePeerDependencies: + - supports-color + dev: false + /@ungap/promise-all-settled@1.1.2: resolution: {integrity: sha512-sL/cEvJWAnClXw0wHk85/2L0G6Sj8UB0Ctc1TEMbKSsmpRosqhwj9gWgFRZSrBr2f9tiXISwNhCPmlfqUqyb9Q==} dev: true @@ -3632,6 +3787,20 @@ packages: array-back: 3.1.0 dev: false + /ansi-escapes@4.3.2: + resolution: {integrity: sha512-gKXj5ALrKWQLsYG9jlTRmR/xKluxHV+Z9QEwNIgCfM1/uwPMCuzVVnh5mwTd+OuBZcwSIMbqssNWRm1lE51QaQ==} + engines: {node: '>=8'} + dependencies: + type-fest: 0.21.3 + dev: false + + /ansi-escapes@6.2.0: + resolution: {integrity: sha512-kzRaCqXnpzWs+3z5ABPQiVke+iq0KXkHo8xiWV4RPTi5Yli0l97BEQuhXV1s7+aSU/fu1kUuxgS4MsQ0fRuygw==} + engines: {node: '>=14.16'} + dependencies: + type-fest: 3.13.1 + dev: false + /ansi-regex@2.1.1: resolution: {integrity: sha512-TIGnTpdo+E3+pCyAluZvtED5p5wCqLdezCyhPZzKPcxvFplEt4i+W7OONCKgeZFT3+y5NZZfOOS/Bdcanm1MYA==} engines: {node: '>=0.10.0'} @@ -3897,6 +4066,20 @@ packages: dev: true optional: true + /ast-types@0.14.2: + resolution: {integrity: sha512-O0yuUDnZeQDL+ncNGlJ78BiO4jnYI3bvMsD5prT0/nsgijG/LpNBIr63gTjVTNsiGkgQhiyCShTgxt8oXOrklA==} + engines: {node: '>=4'} + dependencies: + tslib: 2.5.0 + dev: false + + /ast-types@0.15.2: + resolution: {integrity: sha512-c27loCv9QkZinsa5ProX751khO9DJl/AcB5c2KNtA6NRvHKS0PgLfcftz72KVq504vB0Gku5s2kUZzDBvQWvHg==} + engines: {node: '>=4'} + dependencies: + tslib: 2.5.0 + dev: false + /async-each@1.0.3: resolution: {integrity: sha512-z/WhQ5FPySLdvREByI2vZiTWwCnF0moMJ1hK9YQwDTHKh6I7/uSckMetoRGb5UBZPC1z0jlw+n/XCgjeH7y1AQ==} requiresBuild: true @@ -5339,6 +5522,11 @@ packages: engines: {node: '>=0.10.0'} dev: false + /cli-spinners@2.9.2: + resolution: {integrity: sha512-ywqV+5MmyL4E7ybXgKys4DugZbX0FC6LnwrhjuykIjnK9k8OQacQ7axGKnjDXWNhns0xot3bZI5h55H8yo9cJg==} + engines: {node: '>=6'} + dev: false + /cli-truncate@3.1.0: resolution: {integrity: sha512-wfOBkjXteqSnI59oPcJkcPl/ZmwvMMOj340qUIY1SKZCv0B9Cf4D4fAucRkIKQmsIuYK3x1rrgU7MeGRruiuiA==} engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} @@ -5347,6 +5535,11 @@ packages: string-width: 5.1.2 dev: false + /cli-width@4.1.0: + resolution: {integrity: sha512-ouuZd4/dm2Sw5Gmqy6bGyNNNe1qt9RpmxveLSO7KcgsTnU7RXfsw+/bukWGo1abgBiMAic068rclZsO4IWmmxQ==} + engines: {node: '>= 12'} + dev: false + /cliui@3.2.0: resolution: {integrity: sha512-0yayqDxWQbqk3ojkYqUKqaAQ6AfNKeKWRNA8kR0WXzAsdHpP4BIaOmMAG87JGuO6qcobyW4GjxHd9PmhEd+T9w==} dependencies: @@ -5628,6 +5821,14 @@ packages: resolution: {integrity: sha512-dcKFX3jn0MpIaXjisoRvexIJVEKzaq7z2rZKxf+MSr9TkdmHmsU4m2lcLojrj/FHl8mk5VxMmYA+ftRkP/3oKQ==} dev: false + /cross-fetch@3.1.8: + resolution: {integrity: sha512-cvA+JwZoU0Xq+h6WkMvAUqPEYy92Obet6UdKLfW60qn99ftItKjB5T+BkyWOFWe2pUyfQ+IJHmpOTznqk1M6Kg==} + dependencies: + node-fetch: 2.7.0 + transitivePeerDependencies: + - encoding + dev: false + /cross-spawn@5.1.0: resolution: {integrity: sha512-pTgQJ5KC0d2hcY8eyL1IzlBPYjTkyH72XRZPnLyKus2mBfNjQs3klqbJU2VILqZryAZUt9JOb3h/mWMy23/f5A==} dependencies: @@ -7005,6 +7206,13 @@ packages: tunnel-agent: 0.6.0 dev: false + /figures@3.2.0: + resolution: {integrity: sha512-yaduQFRKLXYOGgEn6AZau90j3ggSOyiqXU0F9JZfeXYhNa+Jk4X+s45A2zg5jns87GAFa34BBm2kXw4XpNcbdg==} + engines: {node: '>=8'} + dependencies: + escape-string-regexp: 1.0.5 + dev: false + /figures@5.0.0: resolution: {integrity: sha512-ej8ksPF4x6e5wvK9yevct0UCXh8TTFlWGVLlgjZuoBH1HwjIfKE/IdL5mq89sFA7zELi1VhKpmtDnrs7zWyeyg==} engines: {node: '>=14'} @@ -9621,6 +9829,15 @@ packages: hasBin: true dev: false + /mute-stream@0.0.8: + resolution: {integrity: sha512-nnbWWOkoWyUsTjKrhgD0dcz22mdkSnpYqbEjIm2nhwhuxlSkpywJmBo8h0ZqJdkp73mb90SssHkN4rsRaBAfAA==} + dev: false + + /mute-stream@1.0.0: + resolution: {integrity: sha512-avsJQhyd+680gKXyG/sQc0nXaC6rBkPOfyHYcFb9+hdkqQkR9bdnkJ0AMZhke0oesPqIO+mFFJ+IdBc7mst4IA==} + engines: {node: ^14.17.0 || ^16.13.0 || >=18.0.0} + dev: false + /mysql@2.18.1: resolution: {integrity: sha512-Bca+gk2YWmqp2Uf6k5NFEurwY/0td0cpebAucFpY/3jhrwrVGuxU2uQFCHjU19SJfje0yQvi+rVWdq78hR5lig==} engines: {node: '>= 0.6'} @@ -9791,6 +10008,18 @@ packages: whatwg-url: 5.0.0 dev: false + /node-fetch@2.7.0: + resolution: {integrity: sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==} + engines: {node: 4.x || >=6.0.0} + peerDependencies: + encoding: ^0.1.0 + peerDependenciesMeta: + encoding: + optional: true + dependencies: + whatwg-url: 5.0.0 + dev: false + /node-forge@1.3.1: resolution: {integrity: sha512-dPEtOeMvF9VMcYV/1Wb8CPoVAXtp6MKMlcbAt4ddqmGqUJ6fQZFXkNZNkNlfevtNkGtaSoXf/vNNNSvgrdXwtA==} engines: {node: '>= 6.13.0'} @@ -9800,6 +10029,13 @@ packages: resolution: {integrity: sha512-O5lz91xSOeoXP6DulyHfllpq+Eg00MWitZIbtPfoSEvqIHdl5gfcY6hYzDWnj0qD5tz52PI08u9qUvSVeUBeHw==} dev: false + /node-localstorage@2.2.1: + resolution: {integrity: sha512-vv8fJuOUCCvSPjDjBLlMqYMHob4aGjkmrkaE42/mZr0VT+ZAU10jRF8oTnX9+pgU9/vYJ8P7YT3Vd6ajkmzSCw==} + engines: {node: '>=0.12'} + dependencies: + write-file-atomic: 1.3.4 + dev: false + /node-releases@2.0.13: resolution: {integrity: sha512-uYr7J37ae/ORWdZeQ1xxMJe3NtdmqMC/JZK+geofDrkLUApKRHPd18/TxtBOJ4A0/+uUIliorNrfYV6s1b02eQ==} @@ -10827,6 +11063,16 @@ packages: dependencies: picomatch: 2.3.1 + /recast@0.21.5: + resolution: {integrity: sha512-hjMmLaUXAm1hIuTqOdeYObMslq/q+Xff6QE3Y2P+uoHAg2nmVlLBps2hzh1UJDdMtDTMXOFewK6ky51JQIeECg==} + engines: {node: '>= 4'} + dependencies: + ast-types: 0.15.2 + esprima: 4.0.1 + source-map: 0.6.1 + tslib: 2.5.0 + dev: false + /redent@3.0.0: resolution: {integrity: sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg==} engines: {node: '>=8'} @@ -11139,6 +11385,16 @@ packages: resolution: {integrity: sha512-R3wLbuAYejpxQjL/SjXo1Cjv4wcJECnMRT/FlcCfTwCBhaji9rWaRCoVEQ1SPiTJ4kKK+yh+bZLAV7SCafoDDw==} dev: false + /run-async@2.4.1: + resolution: {integrity: sha512-tvVnVv01b8c1RrA6Ep7JkStj85Guv/YrMcwqYQnwjsAS2cTmmPGBBjAjpCW7RrSodNSoE2/qg9O4bceNvUuDgQ==} + engines: {node: '>=0.12.0'} + dev: false + + /run-async@3.0.0: + resolution: {integrity: sha512-540WwVDOMxA6dN6We19EcT9sc3hkXPw5mzRNGM3FkdN/vtE9NFvj5lFAPNwUDmJjXidm3v7TC1cTE7t17Ulm1Q==} + engines: {node: '>=0.12.0'} + dev: false + /run-parallel@1.2.0: resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} dependencies: @@ -11228,6 +11484,14 @@ packages: dependencies: lru-cache: 6.0.0 + /semver@7.6.0: + resolution: {integrity: sha512-EnwXhrlwXMk9gKu5/flx5sv/an57AkRplG3hTK68W7FRDN+k+OWBj65M7719OkA82XLBxrcX0KSHj+X5COhOVg==} + engines: {node: '>=10'} + hasBin: true + dependencies: + lru-cache: 6.0.0 + dev: false + /sequin@0.1.1: resolution: {integrity: sha512-hJWMZRwP75ocoBM+1/YaCsvS0j5MTPeBHJkS2/wruehl9xwtX30HlDF1Gt6UZ8HHHY8SJa2/IL+jo+JJCd59rA==} engines: {node: '>=0.4.0'} @@ -11360,6 +11624,10 @@ packages: is-fullwidth-code-point: 4.0.0 dev: false + /slide@1.1.6: + resolution: {integrity: sha512-NwrtjCg+lZoqhFU8fOwl4ay2ei8PaqCBOUV3/ektPY9trO1yQ1oXEfmHAhKArUVUr/hOHvy5f6AdP17dCM0zMw==} + dev: false + /smartwrap@2.0.2: resolution: {integrity: sha512-vCsKNQxb7PnCNd2wY1WClWifAc2lwqsG8OaswpJkVJsvMGcnEntdTCDajZCkk93Ay1U3t/9puJmb525Rg5MZBA==} engines: {node: '>=6'} @@ -12430,6 +12698,11 @@ packages: engines: {node: '>=10'} dev: true + /type-fest@0.21.3: + resolution: {integrity: sha512-t0rzBq87m3fVcduHDUFhKmyyX+9eo6WQjZvf51Ea/M0Q7+T374Jp1aUiyUl0GKxp8M/OETVHSDvmkyPgvX+X2w==} + engines: {node: '>=10'} + dev: false + /type-fest@0.6.0: resolution: {integrity: sha512-q+MB8nYR1KDLrgr4G5yemftpMC7/QLqVndBmEEdqzmNj5dcFOO4Oo8qlwZE3ULT3+Zim1F8Kq4cBnikNhlCMlg==} engines: {node: '>=8'} @@ -12440,6 +12713,11 @@ packages: engines: {node: '>=8'} dev: true + /type-fest@3.13.1: + resolution: {integrity: sha512-tLq3bSNx+xSpwvAJnzrK0Ep5CLNWjvFTOp71URMaAEWBfRb9nnJiBoUe0tF8bI4ZFO3omgBR6NvnbzVUT3Ly4g==} + engines: {node: '>=14.16'} + dev: false + /typed-array-length@1.0.4: resolution: {integrity: sha512-KjZypGq+I/H7HI5HlOoGHkWUUGq+Q0TPhQurLbyrVrvnKTBgzLhIJ7j6J/XTQOi0d1RjyZ0wdas8bKs2p0x3Ng==} dependencies: @@ -12602,6 +12880,11 @@ packages: resolution: {integrity: sha512-jk1+QP6ZJqyOiuEI9AEWQfju/nB2Pw466kbA0LEZljHwKeMgd9WrAEgEGxjPDD2+TNbbb37rTyhEfrCXfuKXnA==} dev: false + /url-join@5.0.0: + resolution: {integrity: sha512-n2huDr9h9yzd6exQVnH/jU5mr+Pfx08LRXXZhkLLetAMESRj+anQsTAh940iMrIetKAmry9coFuZQ2jY8/p3WA==} + engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} + dev: false + /url-parse@1.5.10: resolution: {integrity: sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ==} dependencies: @@ -12870,7 +13153,6 @@ packages: ansi-styles: 4.3.0 string-width: 4.2.3 strip-ansi: 6.0.1 - dev: true /wrap-ansi@7.0.0: resolution: {integrity: sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==} @@ -12880,9 +13162,26 @@ packages: string-width: 4.2.3 strip-ansi: 6.0.1 + /wrap-ansi@8.1.0: + resolution: {integrity: sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==} + engines: {node: '>=12'} + dependencies: + ansi-styles: 6.2.1 + string-width: 5.1.2 + strip-ansi: 7.1.0 + dev: false + /wrappy@1.0.2: resolution: {integrity: sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==} + /write-file-atomic@1.3.4: + resolution: {integrity: sha512-SdrHoC/yVBPpV0Xq/mUZQIpW2sWXAShb/V4pomcJXh92RuaO+f3UTWItiR3Px+pLnV2PvC2/bfn5cwr5X6Vfxw==} + dependencies: + graceful-fs: 4.2.11 + imurmurhash: 0.1.4 + slide: 1.1.6 + dev: false + /write-file-atomic@5.0.0: resolution: {integrity: sha512-R7NYMnHSlV42K54lwY9lvW6MnSm1HSJqZL3xiSgi9E7//FYaI74r2G0rd+/X6VAMkHEdzxQaU5HUOXWUz5kA/w==} engines: {node: ^14.17.0 || ^16.13.0 || >=18.0.0} diff --git a/tools/execute/execute.js b/tools/execute/execute.js new file mode 100644 index 000000000..3d6357275 --- /dev/null +++ b/tools/execute/execute.js @@ -0,0 +1,22 @@ +/** + * Helper function to execute job source in unit tests + */ +import compile from '@openfn/compiler'; +import execute from '@openfn/runtime'; + +export default (job, adaptor, state = {}) => { + // Compile without an adaptor, so there's no import statements + let compiledJob = compile(job) + + // BUT we do need to export the execute function, if present + if (adaptor.execute) { + compiledJob = `export const execute = globalThis.execute; +${compiledJob}` + } + + return execute(compiledJob, state, { + globals: { + ...adaptor, + } + }) +} \ No newline at end of file diff --git a/tools/execute/index.js b/tools/execute/index.js new file mode 100644 index 000000000..1db9a8b3e --- /dev/null +++ b/tools/execute/index.js @@ -0,0 +1,3 @@ +import execute from './execute'; + +export default execute; \ No newline at end of file diff --git a/tools/execute/package.json b/tools/execute/package.json new file mode 100644 index 000000000..60b491166 --- /dev/null +++ b/tools/execute/package.json @@ -0,0 +1,15 @@ +{ + "name": "@openfn/test-execute", + "version": "1.0.0", + "description": "Unit test helper to run jobs from source", + "main": "index.js", + "type": "module", + "private": true, + "keywords": [], + "author": "", + "license": "ISC", + "dependencies": { + "@openfn/compiler": "^0.0.32", + "@openfn/runtime": "^1.0.0" + } +} \ No newline at end of file diff --git a/tutorial.md b/tutorial.md new file mode 100644 index 000000000..b96b40ea4 --- /dev/null +++ b/tutorial.md @@ -0,0 +1,264 @@ +# New Stuff Tutorial + +A nice clean presentation about how I'm thinking about next gen adaptors. + +### Features + +Here's what I've introduced + +* Operation factories +* Operation/implementation split +* Real runtime/job tests +* A pattern for mocking which SHOULD enable a live playground (!!) and maybe easier adaptor creation + +### Motivations + +Here are the problems I'm trying to solve + +* Some sort of capacity for useful unit tests + * They're good for development, maintenance and preventing regressions +* Clarity over what an operation is, and why it matters +* Encouraging good documentation _in the right place_ + * I see lots of problems of devs documenting the wrong functions, wasting time and energy +* (new) If we had a live playground on docs.openfn.org, how would we handle adaptors? + +### Examples + +I've started implementing this stuff in `msgraph` (and common) to see how it all comes together + +I urgently need to start on `salesforce` to explore client-based mocking + +### Issues + +Here's what's not right yet: + +* I worry that this adds formality and complication. Does any of this acutally make adaptor writing easier? +* Expanding references. I'd really like to standardise and simplify this further + - jsdoc path vs dataValue() vs open function + - I am sure that you can just make expand references read '$a.b.c' as a jsonpath +* When the client is abstracted out (like in msgraph), it can be hard to know what it is. You look at `impl.js` and you see request, you don't know what it is. So it's actually kinda hard to use. hmm. +* Maybe the impl pattern makes less sense with the new mocking pattern? Depends how salesforce looks + +### Operation Factories + +Here's an operation factorybeh ar +```js +export const get = operation((state, url, auth, callback) => { + // code goes here + return state +}) +``` + +Why is this good? +- There's less complicated nesting of arrow functions +- It's clear and explicit that this is an Operation + +### Seperate the implementation + +This makes it really easy to write unit tests against an implementation. You just pass a mock client in your tests. + +```js +// operation.js +import { operation } from '@openfn/language-common' +import * as impl from './impl'; + +export const create = operation((state, resource, data, callback) => { + return impl.create(state, request, resource, data, callback) +}) +``` + +Note the extra `request` argument! + +I can now unit test the implementation really cleanly: + +```js +describe('getDrive', () => { + + it('should get a drive by id and set it to state', async () => { + const state = { + configuration: {}, + drives: {} + }; + + // We could add test assertions in the mock if we wanted + const mockRequest = async () => fixtures.driveResponse; + + // Call the implementation with the mock + const result = await impl.getDrive(state, mockRequest, { id: 'b!YXzpkoLwR06bxC8tNdg71m_' }) + + expect(result.drives).to.eql({ default: fixtures.driveResponse }); + }); +}); +``` + +The ability to mock the implementation like this also enables real runtime testing + +### Mocking as first-class adaptor code + +What if each adaptor was able to run in a mock mode. In this mode it will mock out the client entirely and return mock data. A default data suite is included, but can be overridden. + +This works pretty great with undici, I think it should work nicely with client based adaptors too. + +First, the adaptor to expose an `enableMock` function, which is called with state (for config) and optionally some mock data. If mockdata is not provided, defaults will be used. + +Later, the runtime could call `enableMock` in certain circumstances (like a live playground). + +```js +// mock.js (exported by index.js) +export const enableMock = (state, routes = {}) => { + // setup undici to mock at the http level + mockAgent = new MockAgent(); + mockAgent.disableNetConnect(); + + mockPool = mockAgent.get('https://graph.microsoft.com'); + + const mockRoutes = { + ...defaultRoutes, + ...routes + } + + setupMockRoutes() +} + +// name: { pattern, data, options } +const defaultRoutes = { + 'drives': { pattern: /\/drives\//, data: fixtures.driveResponse } +} +``` + +Mocks work based on routing. Obvious with HTTP but I think we can do it with clients too (patterns confirm to` client.`) + +Each adaptor is responsible for implementing its own mock in whatever way makes sense. We will provide strong patterns and utilitie. + +Now, if you want to write unit tests against this mock, you can do so +```js +it('with default mock', async () => { + const state = { ... }; + Adaptor.enableMock(state); + + const result = await Adaptor.getDrive({ id: 'abc' })(state) + expect(result.drives).to.eql({ default: fixtures.driveResponse }) +}) + +it('with custom mock', async () => { + const state = { ... }; + + Adaptor.enableMock(defaultState, { + 'drives': { + pattern: patterns.drives, + data: { x: 22 }, + options: { once: true } + } + }); + + const result = await Adaptor.getDrive({ id: 'abc' })(state) + expect(result.drives).to.eql({ default: fixtures.driveResponse }) +}) +``` + +Important detail: the mock should be considered frozen when activated. If you want to change mock data, call `enable` again + +### Real runtime tests + +I really hate the existing test syntax we use right now. What I actually want to do is write a job and pass it to the actual run time so execution. + +So I've implemented this! + +First, we create an example job in a source file + +```js +// examples.get-drive.js +/** + * A simple test job which gets a drive by id + */ +getDrive(( + state) => ({ id: state.id }), // get the drive id from state + "default", // drive name + (state) => ({ ...state, savedDrives: state.drives }) // alias savedDrives onto state (it gets removed by the adaptor) +) + +``` + +(we can also do this inline in a unit test in a string) + +In a unit test, we can: +* Load this example source +* Load the adaptor module +* Use mock data from above (works great) +* Pass the source, input state and adaptor into the actual runtime and compiler + +That gives us a test that looks like this: +```js +describe('examples', () => { + // setup our mock with whatever data we want + Adaptor.enableMock({ configuration }); + + it('get-drive', async () => { + // Load our example code + const source = loadExample('get-drive') + + // Set up some input state + const state = { + id: 'xxx', + }; + + // Compile and run the job against this adaptor + const finalState = await execute(source, Adaptor, state) + + // Assert on the result state + expect(finalState).to.eql({ ...}); + }) +}); +``` +What's nice about this pattern is that we can run test assertions on the mock handler function as well on the result state + +Note: there is an alternative to all this for real runtime tests. It should be possible to mock out the http requests with undici. I think? Surely at some level of abstraction we can do it. + +I think it's a lot harder to map and mock the different request URLs, but it may be a viable option. + +### Examples + +There's one other benefit of this approach. + +The files in examples.js are just jobs. Regular jobs which you can run through the CLI (given the correct input state and credentials). + +But we should also be able to load these in docusaurus and present them in docs. They're real, runnable, testable execution examples. + +Surley we can take advantage of this? + +### Future Work & Gripes + +Here's some other stuff we need to look at + +* A really good way to "record" a "curl" to the datasource + - I almost see a CLI command `openfn record` which will run a job (probably written against common.http) and save the final `state.data` (or something) to a json file in the right format to be used by the mock. maybe there's a map of `'path.json': data` So you write a job to populate your mock data, then just run it to save the data into the monorepo, then run the unit tests which should run against the mock data. + +* Logging + - Well I've already hooked up and adaptor logger. So console.log should be fine right? Adaptor logging should already be better + - console.success should also be available + +* Replace the export { fn } from 'common' pattern + - Maybe automate it in the build? + +* Reusable functions + - A much broader concern. How do I define a function in my workflow which can be re-used in multiple jobs? + +* Seperate config out + - God I'd love to do this + - `await execute(state, config)` + - `init(config)` + +* What is a breaking change for us? + - Or, a clear policy for what constitutes a major version bump + - Configuration schema is a big sticking point here + +### Documentation + +This is probably a different consideration. But basically I want to rip the doc site up and start again. + +I want a really good, clear, concise, easy to find reference + +I want a few really good, clear, concise, easy to find tutorials + +I want some clear best practice to be outlined. We have this, but it quickly runs stale and isn't very good: https://github.com/OpenFn/adaptors/wiki/Adaptor-Wrting-Best-Practice. It needs to be immediate and obvious. +