I was recently asked by a client to evaluate one of their Node.js systems for areas of improvement. The software made dozens of distinct calls to a few different legacy REST APIs. Each of these calls was developed in isolation, mixed in with business logic. The result was lots of repetitive, duplicative WET code that was hard to follow. The approach drastically increased the code size of the system, making it hard to understand, troubleshoot and modify.
Prior to this, I’ve worked on 3 Node projects over the last few years and I’ve wrapped up all of the REST handling into API Management classes with custom Axios instances. This post summarizes the approach by presenting a simple, clean demo implementation with enough features to show what can be done with Axios customization. Overall, the approach can be applied to other http client libraries as well.
Goals
Here are the overall goals of this effort:
- All API details are dealt with for a given group of related REST endpoints via a single Manager implementation that completely wraps the API
- All REST API details are encapsulated within the Manager implementation
- All non-success scenarios should throw a node.js Error derived object (with embedded Axios error info).
- This should be caught by the caller – we want to avoid tedious nested error handling code or polluting the return values with error information.
- Logging should consistent and reliably performed for each request/response, regardless of the endpoint called
- A correlation id is assigned by the calling process or a random one gets generated.
- The correlation id is consistently passed downstream to the called API
- It should also be consistently and automatically logged (but that is a bit outside of the scope of the demo implementation)
- Note: It would be more convenient to globally provide a correlation id via a mechanism such as a CLS or similar, but that would make the demo harder to follow, so I’ll just stick with the caller passing it in.
- The implementation adheres to the DRY principle: authentication, logging, etc. are consistently handled in one place instead of on a per endpoint basis.
- Responses can be filtered and enriched in a consistent manner, saving callers duplicative work
- In the demo, we will conditionally enrich the content responses with a custom field
Example Rest API
We will need a publicly available API to work against. Ideally, the API should require authentication, have a few endpoints and include different Http methods.
Tinyurl seemed to fit those requirements pretty well. It’s a url shortening service that has been around for a long time. It allows you to register for a free account, and associate aliases (tinyurls) to target urls within your account. They have published Open API docs on their dev site.
In order to keep things from getting too long, we will include the following API endpoints in our demo:
| Method | Path | Description |
| POST | /create | Create a new TinyURL |
| PATCH | /change | Update TinyURLs long (target) URL |
| GET | /alias/{domain}/{alias} | Receive TinyURL information |
| GET | /urls/{type} | Get a list of TinyURLs (available, archived) |
| PATCH | /archive | Archive TinyURL |
This is going to map to our API Manager public methods:
Account Setup
The Tinyurl API requires an apikey to work. To get it you have to register for a free account. Once you’ve done that, you can get your apikey by logging in and clicking on your profile (upper right corner). Select api in the left nav. Fill out the form to create a new apikey by giving it a name and selecting all permissions. It should give you the option to copy it. Save it for use later.
API Manager Structure
The full source code can be seen at the end of this post, or visit the gist
Here we’ll walk through the parts with some running commentary
Constructor
class TinyurlApiManager {
constructor(opts) {
const {baseurl, domain, apikey} = opts;
this._apikey = apikey;
this._domain = domain || "tinyurl.com";
this._axios = axios.create({
validateStatus,
baseURL: (baseurl || "https://api.tinyurl.com/"),
maxContentLength: MAX_CONTENT_LEN,
headers: {
common: {
['Accept']: 'application/json',
['Content-Type']: 'application/json'
}
}
});
Our constructor has only one required parameter: the apikey you got after you setup registration with tinyurl. We save this, and we create a custom axios instance and set a few defaults:
- validateStatus: this is a function we defined to establish what http statuses were considered successful responses and which ones would invoke the error handling.
- baseUrl: all calls should go to the tinyurl api endpoint, so it gets set here once.
- headers: we want these headers in all our requests, so we can set them here once.
this._axios.defaults.headers.common['Authorization'] = `Bearer ${this._apikey}`;
Since each call requires the apikey in the authorization header, we add it here. If this were a token instead, we would have to do it via a request interceptor that would cache the token, check if it is expired and re-fetch it if it is.
Next up we’re going to customize our axios instance with some request and response interceptors. These can be conditionally run before the request is executed or after the response is received.
Axios Request Interceptor
We add the following request interceptor, which runs unconditionally before the request is submitted, to handle the defaulting of the correlation id in cases where it is unset.
this._axios.interceptors.request.use(async config => {
let correlationId = config.headers['x-correlation-id'];
if (correlationId) {
logger.debug("CorrelationId request interceptor: correlationId header already set", { correlationId });
} else {
// correlation id: default to a random value unless it is already set
config.headers = Object.assign(config.headers, {
['x-correlation-id']: uuid.v4()
});
logger.debug("CorrelationId request interceptor: Dynamic headers were added to axios config", {
url: config.url,
headers: filterObject(config.headers, ['x-correlation-id'])
});
}
return config;
});
The filterObject call above just references a simple attribute picker helper function.
Axios Response Interceptors
Next we add our first response interceptor which is intended to demonstrate how response data from the server can be enriched or transformed. The example below is arbitrary. It adds a new calculated enriched_custom_score to the returned object if the alias attribute is returned by the server in the response and the enrich parameter in the original request has a truthy value. The Ramda library is used to cut down on boilerplate property path checking code.
this._axios.interceptors.response.use(
response => {
// if the response payload contains an alias value AND the 'enrich' param is true, add the custom score data to the response
let resAlias = R.path(['data', 'data', 'alias'], response);
let paramEnrich = R.path(['config', 'params', 'enrich'], response);
let doEnrich = !!(resAlias && paramEnrich);
logger.debug("enrichment evaluation", {resAlias, paramEnrich, doEnrich});
if (doEnrich) {
response.data.data.enriched_custom_score = calculateCustomScore(resAlias);
}
logger.debug("post enrichment", {data:response.data});
return response;
});
Our second response interceptor is for logging and error handling. Each successfully completed REST call will emit a log entry with a uniform set of information. This will make troubleshooting easier and increase observability of the application. Failed calls are also logged, and an Error object, augmented with axios error details, is provided to the rejected promise. A fuller implementation would be to subclass Error – this was done in the interest of brevity.
this._axios.interceptors.response.use(
response => {
logger.info("axios call success.", {baseurl: response.config.baseURL, url: response.config.url, status: response.status, statusText: response.statusText});
logger.debug("axios call success payload", {data:response.data});
return response;
},
async error => {
const axiosErrObj = {
url: R.pathOr('-', ['config', 'url'], error),
method: R.pathOr('-', ['config', 'method'], error),
errmsg: error.message,
errstack: error.stack,
errcode: error.code,
response_status: R.pathOr('-', ['response', 'status'], error),
response_body: R.pathOr('-', ['response', 'data'], error)
};
logger.warn("Failed axios call", {error, axiosErrObj});
let e = new Error(`axios error: ${axiosErrObj.errmsg}`);
e.axiosError = axiosErrObj;
return Promise.reject(e);
}
);
That wraps up the constructor and the custom axios instance configuration for this example.
Exec Function
This function is the private, workhorse function that executes the request. It serves as a place where:
- Universal input options are handled, like the addition of the correlation Id to the list of headers
- Validations are run that apply to all requests, like making sure a path is set
- Constructs a consistent response object that will ultimately be used to formulate the response to the public methods
async _execAxios(inOpts) {
// the allowed opts:
const { method, maxContentLength, path, params, data, correlationId } = {...inOpts};
// path must be present in opts
validateTruthy(path, "path must be a specified option in opts for _execAxios call");
const url = path;
const headers = {'x-correlation-id': correlationId};
const config = { method, url, headers, maxContentLength, params, data } ;
logger.debug("_execAxios: Executing an axios call", {config});
const res = await this._axios.request(config);
const retval = {status: res.status, statusText: res.statusText, data: res.data};
logger.debug("_execAxios: Response from axios call", retval);
return retval;
}
Public Methods
These all look pretty cookie cutter. Each one does the following:
- Receives as input an args object with the required arguments for the particular operation (e.g., target url, enrich param, correlation id, etc.)
- Processes those arguments and place them where they need to go (path, body, headers, etc.) in regard to the axios config object which will be used to execute the request
- Performs operation specific validation
- Calls the shared exec function
- Processes the result object and constructs the expected, per-operation response
For example, this is the definition for the createUrl operation (see the other operations in the gist, or in the full source at the end of the post)
/**
* Create a new tinyurl alias
* @param {Object} args
* @param {string} args.url - target url, <b>Required</b>.
* @param {boolean} args.enrich=false - should the response be enriched with custom calculated values? <b>Optional</b>
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, ...}
*/
async createUrl(args) {
const params = filterObject(args, ['enrich']);
const opts = filterObject(args, ['correlationId']);
const body = filterObject(args, ['url']);
const path = '/create';
validateTruthy(body.url, "createUrl: missing required arg: url");
const response = await this._execAxios( {...opts, method: "post", path, params, data: {...body, domain: this._domain}});
const payload = R.path(['data', 'data'], response);
validateTruthy(payload, `createUrl: received empty response from api for url=${body.url}`);
return filterObject(payload, ['alias', 'archived', 'url', 'hits', 'tiny_url', 'enriched_custom_score']);
}
The validateTruthy call above references a helper function for validation.
Other Possible Extensions
The Tinyurl API example is pretty small. It’s big enough to show the basics of how things can be structured and how it can be done using Axios features. It’s quite flexible and I wanted to cover some of the other things I’ve done with it over the last few years.
Retry Logic
Response error interceptors can be used to perform retry logic on failure. This can be either immediately retried for some type of auth failure (perhaps for a token refresh or similar) or saved to a queue and performed at a later time when the backend service is back up.
Response Transformation
Transform response content for all responses by adding a default config.transformResponse function or conditionally based on anything in the request or response via a response interceptor. You could also accomplish this using a request interceptor to trigger the addition of a function to config.transformResponse array dynamically, based on some values of the request or params.
Useful when you want to make response content more uniform with your caller’s standards by converting it in your api manager and hiding the messy details from them.
Request Transformation
Uniformly transform request params to comply with the target API requirements. For example, if the API expects a sort param containing “<SortFieldName> (desc|asc)” (e.g. “lastname desc”), but you want the manager to expose this as two different arguments (sortField and sortOrder) with a default for sortOrder.
You can also transform input data bound for PUT POST or PATCH payloads sent over the API. I have used this to perform serialization of data from what callers have submitted to what the API expects. For example, API vendors sometimes allow ‘custom fields’ or extended functionality, but require only string values when submitting these over the wire to the API. A custom function can be added to the config.transformRequest array to handle this operation – all in one place, keeping things DRY and hiding the complexity from the user.
Full Source Code
See the full code for TinyurlApiManager below, or visit the gist to see the code as well as additional documentation.
const axios = require('axios');
const uuid = require('uuid');
const R = require('ramda');
const winston = require('winston');
const logger = winston.createLogger({
level: 'debug',
transports: [new winston.transports.Console()]
}).child({service: 'TinyurlApiManager'});
const MAX_CONTENT_LEN = 4096;
/**
* Wraps the tinyurl webservice into a concise and easy to use manager object
*/
class TinyurlApiManager {
constructor(opts) {
const {baseurl, domain, apikey} = opts;
this._apikey = apikey;
this._domain = domain || "tinyurl.com";
this._axios = axios.create({
validateStatus,
baseURL: (baseurl || "https://api.tinyurl.com/"),
maxContentLength: MAX_CONTENT_LEN,
headers: {
common: {
['Accept']: 'application/json',
['Content-Type']: 'application/json'
}
}
});
// API key is added to each request. It is static, so we can set here.
// If it were dynamic, we would login, retrieve+cache token in a request interceptor
this._axios.defaults.headers.common['Authorization'] = `Bearer ${this._apikey}`;
logger.info("initialized TinyurlApiManager", {baseurl: this._axios.defaults.baseURL, domain: this._domain});
// Setup Axios Request Interceptors -------------------
// Request Interceptor 1 of 1: Add dynamic headers to all requests
this._axios.interceptors.request.use(async config => {
let correlationId = config.headers['x-correlation-id'];
if (correlationId) {
logger.debug("CorrelationId request interceptor: correlationId header already set", { correlationId });
} else {
// correlation id: default to a random value unless it is already set
config.headers = Object.assign(config.headers, {
['x-correlation-id']: uuid.v4()
});
logger.debug("CorrelationId request interceptor: Dynamic headers were added to axios config", {
url: config.url,
headers: filterObject(config.headers, ['x-correlation-id'])
});
}
return config;
});
// Setup Axios Response Interceptors -------------------
// Response Interceptor 1 of 2: Enrich data when appropriate and the 'enrich' param passed
this._axios.interceptors.response.use(
response => {
// if the response payload contains an alias value AND the 'enrich' param is true, add the custom score data to the response
let resAlias = R.path(['data', 'data', 'alias'], response);
let paramEnrich = R.path(['config', 'params', 'enrich'], response);
let doEnrich = !!(resAlias && paramEnrich);
logger.debug("enrichment evaluation", {resAlias, paramEnrich, doEnrich});
if (doEnrich) {
response.data.data.enriched_custom_score = calculateCustomScore(resAlias);
}
logger.debug("post enrichment", {data:response.data});
return response;
});
// Response Interceptor 2 of 2: Uniform Logging and Error normalization:
// Failed calls return an Error object that contains a custom 'axiosError' attribute.
this._axios.interceptors.response.use(
response => {
logger.info("axios call success.", {baseurl: response.config.baseURL, url: response.config.url, status: response.status, statusText: response.statusText});
logger.debug("axios call success payload", {data:response.data});
return response;
},
async error => {
const axiosErrObj = {
url: R.pathOr('-', ['config', 'url'], error),
method: R.pathOr('-', ['config', 'method'], error),
errmsg: error.message,
errstack: error.stack,
errcode: error.code,
response_status: R.pathOr('-', ['response', 'status'], error),
response_body: R.pathOr('-', ['response', 'data'], error)
};
logger.warn("Failed axios call", {error, axiosErrObj});
let e = new Error(`axios error: ${axiosErrObj.errmsg}`);
e.axiosError = axiosErrObj;
return Promise.reject(e);
}
);
}
// ==================================
// Begin: Public interface
/**
* Fetch a tinyurl with an alias
* @param {Object} args
* @param {string} args.alias - alias identifier of tinyurl, <b>Required</b>.
* @param {boolean} args.enrich=false - should the response be enriched with custom calculated values? <b>Optional</b>
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, ...}
*/
async getUrl(args) {
const params = filterObject(args, ['alias', 'enrich']);
const opts = filterObject(args, ['correlationId']);
const path = `/alias/${this._domain}/${params.alias}`
validateTruthy(params.alias, "getUrl: missing required param: alias");
const response = await this._execAxios( {...opts, path, params});
const payload = R.path(['data', 'data'], response);
validateTruthy(payload, `getUrl: received empty response from api for alias=${params.alias}`);
return filterObject(payload, ['alias', 'archived', 'url', 'hits', 'tiny_url', 'enriched_custom_score']);
}
/**
* Create a new tinyurl alias
* @param {Object} args
* @param {string} args.url - target url, <b>Required</b>.
* @param {boolean} args.enrich=false - should the response be enriched with custom calculated values? <b>Optional</b>
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, ...}
*/
async createUrl(args) {
const params = filterObject(args, ['enrich']);
const opts = filterObject(args, ['correlationId']);
const body = filterObject(args, ['url']);
const path = '/create';
validateTruthy(body.url, "createUrl: missing required arg: url");
const response = await this._execAxios( {...opts, method: "post", path, params, data: {...body, domain: this._domain}});
const payload = R.path(['data', 'data'], response);
validateTruthy(payload, `createUrl: received empty response from api for url=${body.url}`);
return filterObject(payload, ['alias', 'archived', 'url', 'hits', 'tiny_url', 'enriched_custom_score']);
}
/**
* Update the target url of an existing tinyurl alias
* @param {Object} args
* @param {string} args.alias - alias identifier of tinyurl, <b>Required</b>.
* @param {string} args.url - new target url, Required.
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {url, alias}
*/
async updateUrl(args) {
const opts = filterObject(args, ['correlationId']);
const body = filterObject(args, ['url', 'alias']);
const path = '/change';
validateTruthy(R.path(['url'], body), "updateUrl: missing required arg: url");
validateTruthy(R.path(['alias'], body), "updateUrl: missing required arg: alias");
const response = await this._execAxios( {...opts, method: "patch", path, data: {...body, domain: this._domain}});
const payload = R.path(['data', 'data'], response);
validateTruthy(payload, `updateUrl: received empty response from api for alias=${body.alias}`);
// api response is limited
let resp_body = filterObject(payload, ['url']);
return {...resp_body, alias: body.alias};
}
/**
* Archive an existing tinyurl
* @param {Object} args
* @param {string} args.alias - alias identifier of tinyurl, <b>Required</b>.
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, alias, archived}
*/
async archiveUrl(args) {
const opts = filterObject(args, ['correlationId']);
const body = filterObject(args, ['alias']);
const path = '/archive';
validateTruthy(R.path(['alias'], body), "archiveUrl: missing required arg: alias");
const response = await this._execAxios( {...opts, method: "patch", path, data: {...body, domain: this._domain}});
const payload = R.path(['data', 'data'], response);
validateTruthy(payload, `archiveUrl: received empty response from api for alias=${body.alias}`);
return filterObject(payload, ['alias', 'archived', 'tiny_url']);
}
/**
* Retrieve a list of available tinyurls
* @param {Object} args
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, ...}
*/
async getAvailable(args) {
return await this._getList('/urls/available', args);
}
/**
* Retrieve a list of archived tinyurls
* @param {Object} args
* @param {string} args.correlationId - the correlation id under which this should be logged and x-correlation-id header, <b>Optional</b><br/> If missing, a random value is generated.
* @returns {Object} {tiny_url, ...}
*/
async getArchived(args) {
return await this._getList('/urls/archived', args);
}
// End: Public interface
// ==================================
// Begin: Private member helpers
// shared axios method exec function
async _execAxios(inOpts) {
// the allowed opts:
const { method, maxContentLength, path, params, data, correlationId } = {...inOpts};
// path must be present in opts
validateTruthy(path, "path must be a specified option in opts for _execAxios call");
const url = path;
const headers = {'x-correlation-id': correlationId};
const config = { method, url, headers, maxContentLength, params, data } ;
logger.debug("_execAxios: Executing an axios call", {config});
const res = await this._axios.request(config);
const retval = {status: res.status, statusText: res.statusText, data: res.data};
logger.debug("_execAxios: Response from axios call", retval);
return retval;
}
async _getList(path, args) {
const opts = filterObject(args, ['correlationId']);
const response = await this._execAxios( {...opts, path});
const payload = R.path(['data', 'data'], response);
validateIsArray(payload, `getList: received non-array response from api at: ${path}`);
return payload.map(e=>filterObject(e, ['alias', 'archived', 'tiny_url']));
}
// End: Private member helpers
}
module.exports = TinyurlApiManager;
// ==================================
// various private helper functions
// define success status codes
function validateStatus(status) {
return status >= 200 && status < 300;
}
function filterObject(obj, allowedAttributeNames) {
return R.pick(allowedAttributeNames || [], obj || {} );
}
function calculateCustomScore(val) {
// this is just an example to show response data enrichment: - calculate a value between 0-100 for val based on characters in val
// irl this would be more meaningful, but it's just a demo
return (""+(val || ""))
.split("")
.map(x=>x.charCodeAt(0)).reduce((acc, val) => acc+val, 0)
% 101;
}
function validateTruthy(val, msg) {
if (!val) {
throw new Error(msg);
}
}
function validateIsArray(val, msg) {
if (!Array.isArray(val)) {
throw new Error(msg);
}
}
the semi-crazy blog 