README.md

The DATA Broker Application Programming Interface (API)

The DATA Broker API powers the Data Broker's data submission process.

Background

The U.S. Department of the Treasury is building a suite of open-source tools to help federal agencies comply with the DATA Act and to deliver the resulting standardized federal spending information back to agencies and to the public.

For more information about the Data Broker codebase, please visit this repository's main README.

A Note on CGAC/FREC: In the vast majority of cases, top-level agencies identify themselves for purposes of DABS submissions or detached D1/D2 file generation by their 3-digit CGAC code. CGAC are issued and managed by OMB and are updated yearly in the A-11 circular appendix C. The CGAC is equivalent to the treasury concept of the Agency Identifier (AID) embedded in all Treasury Account Symbols (TAS).

In a few cases, legitimately separate (at least for financial reporting purposes) agencies share a CGAC. To allow them to report as separate entities in the Data Broker, we leveraged an internal Treasury element called the Financial Reporting Entity Code (FREC) that Treasury already uses to distinguish between these agencies with shared AID at the TAS level. This field comes from Treasury's CARS system. These agencies, listed in the table below, should use this four-digit FREC code for purposes of identifying themselves in DABS instead of the CGAC they share with one or more agencies. The following is the complete list of agencies supported under the FREC paradigm in DABS. These agencies should always identify themselves to the Broker with the 4-digit FREC code instead of the 3 digit CGAC they share with other agencies.

SHARED CGAC	AGENCY NAME	AGENCY ABBREVIATION	Financial Reporting Entity Code (FREC)
011	EOP Office of Administration	EOPOA	1100
011	Peace Corps	Peace Corps	1125
011	Inter-American Foundation	IAF	1130
011	U.S. Trade and Development Agency	USTDA	1133
011	African Development Foundation	ADF	1136
016	Department of Labor	DOL	1601
016	Pension Benefit Guaranty Corporation	PBGC	1602
033	Smithsonian Institution	SI	3300
033	John F. Kennedy Center For The Performing Arts	Kennedy Center	3301
033	National Gallery of Art	National Gallery	3302
033	Woodrow Wilson International Center For Scholars	Wilson Center	3303
352	Farm Credit Administration	FCA	7801
352	Farm Credit System Insurance Corporation	FCSIC	7802
537	Federal Housing Finance Agency	FHFA	9566
537	Federal Housing Finance Agency Inspector General	FHFAIG	9573

Broker API Project Layout

The Broker API has two major directories: scripts and handlers.

dataactbroker/
├── scripts/        (Install and setup scripts)
└── handlers/       (Route handlers)

Scripts

The /dataactbroker/scripts folder contains the install scripts needed to setup the broker API for a local install. For complete instructions on running your own copy of the API and other Data Broker components, please refer to the documentation in the Data Broker core repository.

Handlers

The dataactbroker/handlers folder contains the logic to handle requests that are dispatched from various routes files. Routes defined in these files may include the @requires_login and @requires_submission_perms tags to the route definition. This tag adds a wrapper that checks if there exists a session for the current user and if the user is logged in, as well as checking the user's permissions to determine if the user has access to this route. If user is not logged in to the system or does not have access to the route, a 401 HTTP error will be returned. This tags are defined in dataactbroker/permissions.py.

account_handler.py contains the functions to check logins and to log users out.

fileHandler.py contains functions for managing user file interaction. It creates all of the jobs that are part of the user submission and has query methods to get the status of a submission. In addition, this class creates downloadable links to error reports created by the Data Broker Validator.

Data Broker Route Documentation

The base URL for all routes is https://broker-api.usaspending.gov/. This should be present ahead of all listed routes in order to use the Broker API (e.g. https://broker-api.usaspending.gov/list_agencies).

All routes that require a login should be passed a header "x-session-id". The value for this header should be taken from the login route response header "x-session-id".

Route documentation is split into folders that reflect the internal organization of routes within the codebase. Each of these sections is detailed below, including a general description of the kinds of routes in each folder and links to documentation for each route in the folder.

All routes have an optional trailing slash, meaning the route will work with or without it.

GET "/"

This route is simply a check to ensure that Broker is running.

Response: "Broker is running"

Dashboard Routes

Dashboard routes are primarily used by the DABS dashboard for analytical displays of warning/error data within a submission or set of submissions.

• active_submission_overview
• active_submission_table
• get_impact_counts
• get_rule_labels
• get_significance_counts
• historic_dabs_graphs
• historic_dabs_table

Domain Routes

Domain routes are primarily used to provide domain information, such as lists of CGAC/FREC agencies.

• list_agencies
• list_all_agencies
• list_data_sources
• list_sub_tier_agencies

File Routes

File routes are primarily routes that are directly related to portions of submissions that are not covered by other subsets of routes, such as the generation routes section.

• certify_dabs_submission
• check_current_page
• check_status
• delete_submission
• get_published_file
• get_submitted_published_file
• get_comments_file
• get_detached_file_url
• get_fabs_meta
• get_file_url
• get_obligations
• get_submission_comments
• get_submission_zip
• latest_publication_period
• list_banners
• list_history
• list_submissions
• list_latest_published_files
• publish_and_certify_dabs_submission
• publish_dabs_submission
• publish_fabs_file
• published_submissions
• report_url
• restart_validation
• revalidation_threshold
• revert_submission
• submission_data
• submission_metadata
• update_submission_comments
• upload_dabs_files
• upload_fabs_file

Generation Routes

Generation routes are used for generating files (A, D1, D2, E, and F) and checking on the status of generations that have been started.

• check_detached_generation_status
• check_generation_status
• generate_detached_file
• generate_file

Login Routes

Login routes are used to log a user in or out or check if the current session is still active.

• login
• logout
• caia_login
• session

Settings Routes

Settings routes are used to set agency-wide settings for the DABS Dashboard. These only affect the dashboard display and will have no bearing on actual submissions.

• rule_settings
• save_rule_settings

User Routes

User routes are used to get information about available users or get/set information about the currently logged in user.

• active_user
• list_submission_users
• list_user_emails
• set_skip_guide

Automated Tests

Many of the broker tests involve interaction with a test database. However, these test databases are all created and torn down dynamically by the test framework, as new and isolated databases, so a live PostgreSQL server is all that's needed.

These types of tests should all be filed under the data-act-broker-backend/tests/integration folder, however the reality is that many of the tests filed under data-act-broker-backend/tests/unit also interact with a database.

So first, ensure your dataactcore/local_config.yml and dataactcore/local_secrets.yml files are configured to be able to connect and authenticate to your local Postgres database server as instructed in INSTALL.md

To run all tests

$ pytest

To run just integration tests

$ pytest tests/integration/*

To run just Broker API unit tests

$ pytest tests/unit/dataactbroker/*

To generate a test coverage report with the run, just append the --cov flag to the pytest command.