- Table of contents
- 1. Introduction
- 2. How to build
- 3. Dependencies
- 4. Endpoints
- 5. Caching
- 6. Testing
- 7. Edge cases
- 8. Changes after deadline
The goal of assignment 2 was to create a service that gives information about the renewable energy status for countries. The service offers two endpoints that serves the current and historic data respectively. These endpoints are:
energy/v1/renewables/history/
energy/v1/renewables/current/
Alongside the two endpoints, it also supports the registration of webhooks which it
stores in a firestore database.
Additionally, the service also has a status endpoint which is used to present the
diagnostics of the service.
The endpoints for these two services are:
energy/v1/notifications/
energy/v1/status/
Rebuild and run docker in the background:
sudo docker compose build
sudo docker compose up -d
To run the application, you also need to run a firestore database in the background.
The authentication file needs to be named firestoreauth.json. The file needs to be put in a
folder like this authentication/firestoreauth.json from the root directory.
The application will now run on port 8080
This endpoint uses one other third party service for information collection.
This service is the rest countries api which is used for finding borders of countries.
For this project we use a NTNU internal version of the api found at the address:
http://129.241.150.113:8080/
The service also relies on Firestore for data persistence. This database is used for webhook persistence and caching of the third party api.
Country information is stored internally as a csv file. This was chosen because putting the data on firebase would lead to an unnecessary load on the firebase storage. In addition, the file is relatively lightweight so bundling with the project has a low weight cost.
This section will give an introduction into what the endpoints do and how to use them. If nothing is found for a request an empty array is returned.
Current renewables is an endpoint that serves to give the user the "current" status of renewable energy in a given country. Current is defined as the most recent entries into the dataset, which is currently 2021.
An example request for a given country can be:
energy/v1/renewables/current/{country code/country name}
which gives the response in json format as:
[
{
"Name": "Norway",
"Code": "NOR",
"Year": 2021,
"Renewable": 71.558365
}
]This endpoint can also be queried to showcase the country's neighbours alongside it
using requests such as:
energy/v1/renewables/current/{country code/country name}?neighbours=true
which gives the following response:
[
{
"Name": "Norway",
"Code": "NOR",
"Year": 2021,
"Renewable": 71.558365
},
{
"Name": "Finland",
"Code": "FIN",
"Year": 2021,
"Renewable": 34.61129
},
{
"Name": "Sweden",
"Code": "SWE",
"Year": 2021,
"Renewable": 50.924007
},
{
"Name": "Russia",
"Code": "RUS",
"Year": 2021,
"Renewable": 6.6202893
}
]If a country is not in the dataset, it will not be returned even if its neighbours are.
In other words a query with neighbours=true on a country not in the dataset, will only
return the neighbour countries.
History renewables is an endpoint that enables queries of a country's historic renewable status.
The endpoint can be given requests for specific countries or for the average of all countries.
An example request for a given country can be:
energy/v1/renewables/history/{country}
which gives the response in json format as:
[
{
"Name": "Norway",
"Code": "NOR",
"Year": 1965,
"Renewable": 67.87996
},
{
"Name": "Norway",
"Code": "NOR",
"Year": 1966,
"Renewable": 65.3991
},
...
{
"Name": "Norway",
"Code": "NOR",
"Year": 2021,
"Renewable": 71.558365
}
]If no country is specified, the method returns the average for all countries in the format:
[
{
"name": "Chile",
"isoCode": "CHL",
"percentage": 21.68958424561404
},
{
"name": "Germany",
"isoCode": "DEU",
"percentage": 4.983359357894738
},
{
"name": "Netherlands",
"isoCode": "NLD",
"percentage": 1.7173033947438596
},
...
]The endpoint also supports a range of different queries.
begin and end can be used to specify timeframes for history, for average countries
this will restrict where the average is calculated. These can be used together or separated with
?begin=year&end=year. If a country that is in the dataset does not have an entry in this
time slice, the renewable result will be:"percentage": -1, signaling no data.
The output can also be sorted by value by using sortByValue={bool}
Status is an endpoint that enables queries on the availability of all individual services the server depends on.
An example request for the status can be:
energy/v1/status/
which gives the response in json format as:
{
"countriesapi": 200,
"notification_db": 200,
"webhooks": 1,
"version": "V1",
"uptime": 2569.4878355
}The status codes and the amount of webhooks can change depending on the status of the services and the amount of webhooks that are up.
There are two available webhook events, Country and Endpoint. The country event is triggered each time a country is invoked. The endpoint event is triggered on when the endpoints in the service is invoked. The country can be registered on alpha-3 code.
The notification endpoint serves the purpose of allowing users to register webhooks to get notifications when endpoints are called. The endpoint also allows users to delete webhooks and request info for a given webhook. The event type can be specified in optional parameter eventType. If no event type is given the country event will be assumed.
For example to specify that you want to use the event type endpoint.
energy/v1/notifications?eventType=endpoint
The body will be identical except that country can be omitted.
Note that for GET and POST the evenType has to be specified if endpoint event is wanted.
This is not needed for DELETE.
To create a new webhook, the user needs to send a POST request with a json object with the form:
{
"url": "example.com",
"country": "NOR",
"calls": 5
}This request will be answered with the id of the webhook on the server:
{
"webhook_id": "somerandomstring"
}If the user submits unnecessary content the server silently discards it
To delete a webhook the user only needs to send a DELETE request with the form:
energy/v1/notifications/{id}"
To view a registered webhook the user can send a GET request with the form:
energy/v1/notifications/{id},
which will give the following response if the webhook was found:
{
"url": "Sample.com",
"country": "NOR",
"calls": 5
}On a webhook TRIGGER the webhook will send a payload to the webhook url. The message will be a json post request with the form:
{
"webhook_id": "somerandomidstring",
"country": "nor",
"calls": 1
}To avoid unnecessary requests for the external country api, the borders of countries are cached on firestore. If a cached item is older than 24 hours, the item will be replaced when you try and retrieve the item. The disadvantage of this is that the firestore read/write is increased.
To avoid unnecessary requests, all the tests implement stubbing of the third party api. This is done in the application through dependency injection for the requests.
Methods that do http requests, do this through an interface that is passed into it. For the tests, a "stubbed" implementation of this interface is used instead of the actual http request. In practise this means using a function that returns some predetermined data for a given http request url.
Because the firestore database is not considered a third party, this service is not stubbed. To separate test calls from real calls, we use test collections in the database for our service. These test collections are utilized by passing in the database handler with a different collection into the functions during tests. For certain tests that are sensitive to the fire base state, a randomized collection is used which is torn down afterwards
Since we use a third-party library to find country neighbours, even if we don't have the country in our dataset, there are certain "invalid" queries that goes through.
An example of this is querying on ger which is not a valid name nor iso-code,
but the rest countries api still returns results for germany for this. This also gives
the api the property that e.g. germany will be found for the search ger only if
neighbours=true
There are some tests that rely on the firestore updating. These tests integrate a built-in delay to give the messages time to arrive. They have a generous delay, but they might fail in extreme circumstances even if the code is correct.
Changed line 202 in currentRenewables from:
pathParts := strings.Split(strings.TrimSuffix(r.URL.Path, "/"), "/")
back to an older state which was:
pathParts := strings.Split(r.URL.Path, "/")
This fixes a bug that does not allow for all countries to be found.
There are better fixes available but out of interest in keeping the code as "unchanged" as possible, we chose to do this instead.