Merge pull request #99 from vlocityinc/feature/build_manifest

Update Readme to include Manifest info.
Add ability to build manifest as a command.
Better test for breaking JSON responses.
Updates for better error handling
Show more consistent time.
Add "key" as argument alternative to manifest style.

Author: Adam Rutland

README.md

@@ -51,11 +51,12 @@ Authentication with Salesforce DX credentials is now possible. Use `-sfdx.userna
 * [All Commands](#all-commands)
   * [Example Commands](#example-commands)
   * [Additional Command Line Options ](#additional-command-line-options)
-* [Advanced Job File Settings](#advanced-job-file-settings)
+* [Developer Workflow](#developer-workflow)
+* [Other Job File Settings](#other-job-file-settings)
 * [Supported DataPack Types](#supported-datapack-types)
 * [Advanced](#advanced)
   * [Matching Keys](#matching-keys)
-* [OmniOut](#OmniOut)
+* [OmniOut](#omniout)
 
 # Installation Instructions
 -----------
@@ -520,17 +521,17 @@ This will provide a list of files that are different locally than in the org. In
 -----------
 
 ## Primary  
-`packExport`: Export from a Salesforce org into a DataPack Directory  
-`packExportSingle`: Export a Single DataPack by Id  
-`packExportAllDefault`: Export All Default DataPacks as listed in Supported Types Table  
-`packDeploy`: Deploy all contents of a DataPacks Directory  
+`packExport`: Export from a Salesforce org into a DataPack Directory    
+`packExportSingle`: Export a Single DataPack by Id    
+`packExportAllDefault`: Export All Default DataPacks as listed in Supported Types Table    
+`packDeploy`: Deploy all contents of a DataPacks Directory    
 
 ## Troubleshooting 
 `packContinue`: Continues a job that failed due to an error  
-`packRetry`: Continues a Job retrying all deploy errors or re-running all export queries  
-`validateLocalData`:  Check for Missing Global Keys in Data.  
-`cleanOrgData`: Run Scripts to Clean Data in the Org and Add Global Keys to SObjects missing them  
-`refreshProject`: Refresh the Project's Data to the latest format for this tool  
+`packRetry`: Continues a Job retrying all deploy errors or re-running all export queries    
+`validateLocalData`:  Check for Missing Global Keys in Data.    
+`cleanOrgData`: Run Scripts to Clean Data in the Org and Add Global Keys to SObjects missing them    
+`refreshProject`: Refresh the Project's Data to the latest format for this tool    
 
 ## Additional 
 `packGetDiffsAndDeploy`: Deploy only files that are modified compared to the target Org  
@@ -667,7 +668,99 @@ The Job file additionally supports some Vlocity Build based options and the opti
 | type | DataPack Type used for packExportSingle command | String | none |
 | verbose | Show additional logging statements | Boolean | false |
 
-# Advanced Job File Settings
+# Developer Workflow
+------
+When developing on a large project, exporting DataPacks through Queires is not the ideal process. Instead, each developer should keep track of the major items that they are working on and extract those items as needed to commit to Version Control.
+
+## Manifest Driven Workflow
+The `manifest` section of the Job File can be used to create a collection of DataPacks to be exported or deployed. Manifests are based on the VlocityDataPackKey of each DataPack.
+
+### VlocityDataPackKey Overview
+Each DataPack has a VlocityDataPackKey that represents its Unique Name in the DataPacks system. After Exporting a DataPack this VlocityDataPackKey is also the `VlocityDataPackType/FolderName` of the DataPack. 
+
+For Product2 DataPacks, the VlocityDataPackKey is built using the  `vlocity_namespace__GlobalKey__c`. Given the Product2:
+``` json
+{
+    "%vlocity_namespace%__GlobalKey__c": "7e0b4fa5-4290-d439-ba96-f1a1c3708b0b",
+    "Family": "Phones",
+    "IsActive": true,
+    "Name": "iPhone 6s Plus",
+    "VlocityDataPackType": "SObject",
+    "VlocityRecordSObjectType": "Product2"
+}
+```
+
+The VlocityDataPackKey would be `Product2/7e0b4fa5-4290-d439-ba96-f1a1c3708b0b`. This is also the same as the Folder Name of the Product2 DataPack when saved to the file system. This is because the `GlobalKey__c` of the Product2 is meant to be a unique field, and the Folders must also have unique names.
+
+Here is the `iPhone 6s Plus` Product2 DataPack in its folder with the same VlocityDataPackKey as shown above:
+
+![iPhone6sPlus](doc/iPhone6sPlus.png)
+
+### Using the VlocityDataPackKey in the Manifest
+This VlocityDataPackKey can be added to the `manifest` of the Job File to drive Export of this Product along with all its dependencies.
+
+```yaml
+projectPath: .
+manifest:
+- Product2/7e0b4fa5-4290-d439-ba96-f1a1c3708b0b 
+```
+
+You can see all the available VlocityDataPackKeys for your Manifest by running:  
+`vlocity packGetAllAvailableExports --nojob`
+
+![GetAllOutput](doc/GetAllOutput.png)
+
+This will additionally create the file `VlocityBuildLog.yaml` which will save all the results in a format that you can move to your job file.
+
+```yaml
+manifest:
+  - AttributeCategory/CLOUD_SERVICES
+  - AttributeCategory/MOBILE_HANDSET
+  - AttributeCategory/INTERNET
+  - AttributeCategory/MOBILE_CONTRACT
+  - AttributeCategory/MOBILE_TRADE_IN
+  - AttributeCategory/SIM_CARD
+  - AttributeCategory/WATCH
+  - CalculationMatrix/AgeRating
+  - DataRaptor/Billing Import Bundle
+```
+
+### Adding to Job Files
+When defining your project through a Manifest, it is best to create 2 Job Files.
+
+1. Export Job File
+2. Deploy Job File
+
+#### Export Job File
+The Export Job File will be used to *only* export the listed manifest items plus their dependencies. For someone only working on a set of Products, this Job File would look like: 
+
+**Export.yaml**
+```yaml
+projectPath: .
+expansionPath: vlocity
+manifest:
+- Product2/1b1b6c6f-c1b0-1c1d-e068-4651202cb07b
+- Product2/6e28982d-57c8-5bd3-e6db-d66c02c20e43
+- Product2/7e0b4fa5-4290-d439-ba96-f1a1c3708b0b
+- Product2/9a86f5d9-2e05-ef0d-3492-3479db233ef8
+```
+
+Running `packExport -job Export.yaml` would also export any DataPacks that these Products depend on by default. To only export the exact manifest you can use `-depth 0` to your command. 
+
+#### Deploy Job File
+The manifest is also used to control what is deployed, but in the deploy case it will only deploy waht is specified. This allows deploying only a specific set of local folders to the Org, as the VlocityDataPackKeys are the Folder Paths of the DataPacks.
+
+Therefore, it is best to create a second job file for Deploy. This job file only needs projectPath as it will deploy the entire contents of the project by default.
+
+**Deploy.yaml**
+``` yaml
+projectPath: .
+expansionPath: vlocity
+```
+
+Running `packDeploy -job Deploy.yaml` will then deploy all of the DataPacks in the `vlocity` folder as there is no manifest being defined.
+
+# Other Job File Settings
 ------------
 The Job File has a number of additonal runtime settings that can be used to define your project and aid in making Exports / Deploys run successfully. However, the Default settings should only be modified to account for unique issues in your Org. 
 
@@ -789,6 +882,7 @@ These types are what would be specified when creating a Query or Manifest for th
 | OrchestrationPlanDefinition | OrchestrationPlanDefinition__c |
 | Pricebook2<br>(Salesforce Standard Object) | Pricebook2 |
 | PriceList | PriceList__c<br>PricingElement__c<br>PricingVariable__c<br>PricingVariableBinding__c |
+| PricingPlan | PricingPlan__c<br>PricingPlanStep__c |
 | PricingVariable | PricingVariable__c |
 | Product2<br>(Salesforce Standard Object) | Product2<br>PricebookEntry<br>AttributeAssignment__c<br>ProductChildItem__c<br>OverrideDefinition__c<br>ProductConfigurationProcedure__c<br>ProductRelationship__c<br>ProductEligibility__c<br>ProductAvailability__c<br>RuleAssignment__c<br>ProductRequirement__c<br>ObjectFieldAttribute__c<br>PricingElement__c<br>PriceListEntry__c<br>DecompositionRelationship__c<br>OrchestrationScenario__c | 
 | Promotion | Promotion__c<br>PromotionItem__c |
@@ -870,37 +964,6 @@ module.exports = function(vlocity, currentContextData, jobInfo, callback) {
 });
 ```
 
-##### Export by Manifest
-If Exporting with a Manifest, each JSON Object will be one entry in the Manifest in the form of:
-```yaml
-manifest:
-  VlocityCard: 
-    - Campaign-Story 
-```
-```json
-{
-    "VlocityDataPackType": "VlocityCard",
-    "Id": "Campaign-Story"
-}
-```
-"Id" is the default JSON key in the Manifest, but Manifests also support YAML Key/Value pair syntax:
-```yaml
-manifest: 
-  OmniScript: 
-    - Type: Insurance 
-      SubType: Billing
-      Language: English
-```
-Which becomes:
-```json
-{
-    "VlocityDataPackType": "OmniScript",
-    "Type": "Insurance",
-    "SubType": "Billing",
-    "Language": "English"
-}
-```
-
 ##### Export by Queries
 For a Query, each result from the Query will be a JSON Object with the appropriate DataPack Type.
 ```yaml

codeship/publish.sh

@@ -1,24 +1,24 @@
 #!/usr/bin/env bash 
 set -e
 
-./codeship/decryptFiles.sh
-
-cp codeship/unencrypted_files/npmrc .npmrc
-
-if [ $CI_BRANCH == "master" ]; then
-    npm publish .
-else 
-    npm publish . --tag="next"
-fi
-
 npm run-script build
 
 P_VERSION=`cat package.json | jq -r '. | .version'` 
 
 GITHUB_ASSETS="dist/vlocity-linux-x64,dist/vlocity-macos-x64,dist/vlocity-win-x64.exe,dist/vlocity-win-x86.exe"
 
+./codeship/decryptFiles.sh
+
 if [ $CI_BRANCH == "master" ]; then
     publish-release --notes "$P_VERSION" --token $GITHUB --target_commitish $CI_BRANCH --owner vlocityinc --repo vlocity_build --name "v$P_VERSION" --tag "v$P_VERSION" --assets "$GITHUB_ASSETS" --draft
 else 
     publish-release --notes "Beta - v$P_VERSION" --token $GITHUB --target_commitish $CI_BRANCH --prerelease --reuseRelease --owner vlocityinc --repo vlocity_build --name "Beta-v$P_VERSION" --editRelease --tag next --assets "$GITHUB_ASSETS" --draft
+fi
+
+cp codeship/unencrypted_files/npmrc .npmrc
+
+if [ $CI_BRANCH == "master" ]; then
+    npm publish .
+else 
+    npm publish . --tag="next"
 fi

codeship/test.sh

@@ -1,7 +1,6 @@
 #!/usr/bin/env bash 
 set -e
 
-# Will recreate all files in codeship/unencrypted_files
 ./codeship/decryptFiles.sh
 
 SF_AUTH_ORG=`sfdx force:auth:sfdxurl:store -f codeship/unencrypted_files/test.sfdx --json`
@@ -11,4 +10,17 @@ npm run-script unitTest
 
 npm link
 
-vlocity -sfdx.username $SF_USERNAME runTestJob
+vlocity -sfdx.username $SF_USERNAME runTestJob
+
+# Must return a JSON with a result
+vlocity -sfdx.username $SF_USERNAME runTestJob --json | jq .
+
+npm run-script build
+
+./dist/vlocity-linux-x64 -sfdx.username $SF_USERNAME --nojob packExport -key VlocityCard/datapacktest-card
+
+./dist/vlocity-linux-x64 -sfdx.username $SF_USERNAME --nojob packGetAllAvailableExports --json | jq .
+
+./dist/vlocity-linux-x64 -sfdx.username $SF_USERNAME -projectPath "vlocity-temp" --nojob packExport -key VlocityCard/datapacktest-card
+
+./dist/vlocity-linux-x64 -sfdx.username $SF_USERNAME -projectPath "vlocity-temp" --nojob packDeploy -key VlocityCard/datapacktest-card --json | jq .

doc/GetAllOutput.png

@@ -1,8 +1,4 @@
 module.exports = function(vlocity, dataPackData, jobInfo, callback) {
-
-    //console.log(dataPackData);
     delete dataPackData.VlocityDataPackData.Product2[0].PricebookEntry;
-
-   // console.log(JSON.stringify(dataPackData));
     callback();
 }

doc/iPhone6sPlus.png

@@ -40,19 +40,19 @@ module.exports = function(vlocity, dataPackData, jobInfo, callback) {
                             var currentTime = Date.now();
 
                             vlocity.jsForceConnection.apex.post('/' + vlocity.namespace + '/v2/DataRaptor/', body, function(err, result) {
-                                if (err) { return console.error(err); }
+                                if (err) { return VlocityUtils.error(err); }
 
                                 var thisTime = (Date.now() - currentTime);
 
-                                console.log(dataRaptorName + ' - ' + ((isRolledBack == "true") ? 'Old' : 'New') +': ' + result.ACTUAL + ' CPU: ' + result.CPU);
+                                VlocityUtils.log(dataRaptorName + ' - ' + ((isRolledBack == "true") ? 'Old' : 'New') +': ' + result.ACTUAL + ' CPU: ' + result.CPU);
 
                                 totalCpuTime += result.CPU;
                                 totalTime += result.ACTUAL;
 
                                 timingCallback();
                             });
                         }, function(err, result) {
-                                console.log(dataRaptorName + ' - Final Total: ' + totalTime + ' Final Avg: ' + (totalTime / TOTAL_TO_RUN) + ' CPU ' +  (totalCpuTime / TOTAL_TO_RUN));
+                            VlocityUtils.log(dataRaptorName + ' - Final Total: ' + totalTime + ' Final Avg: ' + (totalTime / TOTAL_TO_RUN) + ' CPU ' +  (totalCpuTime / TOTAL_TO_RUN));
                                 seriesCallback();
                         });
                     });

javascript/removeStandardPricebokEntry.js

@@ -399,42 +399,44 @@ DataPacksBuilder.prototype.initializeImportStatus = function(importPath, jobInfo
             if (dataPackLabel == null || !self.vlocity.datapacksutils.fileExists(metadataFilename)) {
                 return;
             }
-                                
-            self.loadFilesAtPath(path.join(dataPackTypeDir, dataPackName), jobInfo, dataPackKey);
-            var sobjectData = JSON.parse(self.getFileData(metadataFilename));
-            var generatedDataPackKey = dataPackType + '/' + self.vlocity.datapacksexpand.getDataPackFolder(dataPackType, sobjectData.VlocityRecordSObjectType, sobjectData);
+            try {  
+                self.loadFilesAtPath(path.join(dataPackTypeDir, dataPackName), jobInfo, dataPackKey);
+                var sobjectData = JSON.parse(self.getFileData(metadataFilename));
+                var generatedDataPackKey = dataPackType + '/' + self.vlocity.datapacksexpand.getDataPackFolder(dataPackType, sobjectData.VlocityRecordSObjectType, sobjectData);
 
-            jobInfo.keysToDirectories[generatedDataPackKey] = dataPackType + '/' + dataPackName;
+                jobInfo.keysToDirectories[generatedDataPackKey] = dataPackType + '/' + dataPackName;
 
-            if (self.isInManifest(jobInfo, dataPackType, generatedDataPackKey, dataPackName)) {
-            
-                if (!jobInfo.currentStatus[generatedDataPackKey]) {
-                    jobInfo.currentStatus[generatedDataPackKey] = 'Ready';
-                }
+                if (self.isInManifest(jobInfo, dataPackType, generatedDataPackKey, dataPackName)) {
+                
+                    if (!jobInfo.currentStatus[generatedDataPackKey]) {
+                        jobInfo.currentStatus[generatedDataPackKey] = 'Ready';
+                    }
 
-                if (sobjectData.Name) {
-                    jobInfo.generatedKeysToNames[generatedDataPackKey] = sobjectData.Name;
-                }
+                    if (sobjectData.Name) {
+                        jobInfo.generatedKeysToNames[generatedDataPackKey] = sobjectData.Name;
+                    }
 
-                if (jobInfo.allParents.indexOf(generatedDataPackKey) == -1) {
-                    jobInfo.allParents.push(generatedDataPackKey);
-                }
+                    if (jobInfo.allParents.indexOf(generatedDataPackKey) == -1) {
+                        jobInfo.allParents.push(generatedDataPackKey);
+                    }
 
-                var apexImportData = {};
+                    var apexImportData = {};
 
-                self.vlocity.datapacksutils.getApexImportDataKeys(sobjectData.VlocityRecordSObjectType).forEach(function(field) {
-                    apexImportData[field] = sobjectData[field];
-                });
+                    self.vlocity.datapacksutils.getApexImportDataKeys(sobjectData.VlocityRecordSObjectType).forEach(function(field) {
+                        apexImportData[field] = sobjectData[field];
+                    });
 
-                apexImportData.VlocityRecordSourceKey = sobjectData.VlocityRecordSourceKey;
-                jobInfo.preDeployDataSummary.push(apexImportData);
-                jobInfo.allDataSummary[generatedDataPackKey] = apexImportData;
-                jobInfo.dataPackKeyToPrimarySourceKey[sobjectData.VlocityRecordSourceKey] = generatedDataPackKey;
+                    apexImportData.VlocityRecordSourceKey = sobjectData.VlocityRecordSourceKey;
+                    jobInfo.preDeployDataSummary.push(apexImportData);
+                    jobInfo.allDataSummary[generatedDataPackKey] = apexImportData;
+                    jobInfo.dataPackKeyToPrimarySourceKey[sobjectData.VlocityRecordSourceKey] = generatedDataPackKey;
+                }
+            } catch (e) {
+                jobInfo.currentStatus[dataPackKey] = 'Error';
+                jobInfo.hasError = true;
+                jobInfo.errors.push(`Error Loading >> ${dataPackKey} - ${e.message}`);
+                VlocityUtils.error('Error Loading', dataPackKey, e.message);
             }
-            //         }
-            // } catch (e) {
-            //     VlocityUtils.error('Error Initializing', dataPackKey, e.stack);
-            // }
         });
     });
 
@@ -446,7 +448,7 @@ DataPacksBuilder.prototype.initializeImportStatus = function(importPath, jobInfo
             // Ignore when passed in as type
             if (key.indexOf('/') != -1 && !jobInfo.manifestFound[key]) {
                 jobInfo.hasError = true;
-                jobInfo.errors.push('Manifest Item Missing >> ' + key)
+                jobInfo.errors.push('Manifest Item Missing >> ' + key);
                 VlocityUtils.error('Manifest Item Missing', key);
             }
         });