Implementing API data caching

[thedgbrt]

Hi guys,
What would be the best way to handle caching the results of my API calls? I've been playing with the ApiClient.js file (in src/helpers) and tried to replace superagent with superagent-cache, but as far as I can tell it has compatibility issue (see the issue I opened there jpodwys/superagent-cache#21 ).

Has anybody successfully implemented caching with this boilerplate?

[isilweo]

Hi,

we've implemented api cache using memory-cache and writing simple cachedRequest function that we used in api/actions (see below). It is very simple one, it is set to 60 seconds as you can see, in other part of app we have also webhook to invalidate this cache so we could probably switch it to more than 60 sec. We've also implemented cache for html itself if you're intrested i can also post that. in normall production world you would use some reverseproxy anyway

import request from 'request';
import config from '../../src/config.js';
import cache from 'memory-cache';

export default function cachedRequest(url) {
  return new Promise((resolve, reject) => {
    let cachedResponse = cache.get(url);

    if (cachedResponse) {
      resolve(cachedResponse);
    } else {
      request({
        url: url,
        json: true,
      }, (error, response, body) => {
        if (error) {
          reject(error);
        }
        else if (response.statusCode !== 200) {
          reject ({'statusCode': response.statusCode});
        }
        else {
          cache.put(url, body, 60*1000);
          resolve(body);
        }
      });
    }

  });
}

and somewhere in api/actions just use like this

export default function list(req, params) {
  return cachedRequest(`http://${config.dbHost}/banners/${params[0]}`);
}

[thedgbrt]

@isilweo thanks! If I understand correctly in your case caching is implemented for use with the custom api server in /api. What would have to change to make it work with an external API ? I followed these steps to set it up.

I've tried adding it to src/helpers/ApiClient.js but can't wrap my head around it.

[isilweo]

As i understand this project correctly you should always use /api proxy and then download data from external api using actions in api/actions. So it works like this

1. use redux action to ask for data from api - example here https://github.com/erikras/react-redux-universal-hot-example/blob/master/src/redux/modules/info.js#L42
2. [few things happen here]
3. it goes to api/actions/loadInfo.js
   So in here instead

export default function loadInfo() {
  return new Promise((resolve) => {
    resolve({
      message: 'This came from the api server',
      time: Date.now()
    });
  });
}

you should do something like

import cachedRequest from '../../utils/cachedRequest';

export default function list(req, params) {
  return cachedRequest(`http://somehost.com/api/info`);
}

you can also change somehost.com to your own config var and put it into src/config.js

[mmahalwy]

@dagobertrenouf i think there are two places that make good candidates for caching:

1. ApiClient
2. something to do with your proxy
3. the /api route

Of those, I think the last is the best if you are not doing server side rendering. You have to setup a cache here: https://github.com/erikras/react-redux-universal-hot-example/blob/master/src/server.js#L38 where your code would either call from cache or resort to using the proxy to call API, put into cache and return results.

If you are using server-side rendering, you may find libraries out there to help with caching and proxy, hence why point 2. You can have callbacks on the proxy cycle, see: https://github.com/nodejitsu/node-http-proxy#listening-for-proxy-events

You may find ways to cache the superagent call but I don't recommend it as it would act differently on the server and the client (at least I believe so) and caching on the proxy level makes more sense.

At my current job we have this very challenge too and what I am recommending is using a layer between the frontend app and the api app, something like varnish. I don't want my frontend, although it's based on nodejs express server, to handle writing and loading from cache.

[thedgbrt]

@isilweo I don't think you understand the purpose of the /api folder. It represents a demo source of API data, but it has nothing to do with the proxy. The proxy is created with a couple of lines in src/server.js. They have the same name but they work independently. As a proof of that, I keep making proxied api calls (to my own api) and I have deleted the whole /api folder for a long time!

@mmahalwy Thanks for taking the time. My goal is really server side rendering. I agree that having an intermediate layer would be ideal, however I don't have the chops to do it, so I'm resorting to the nasty approach.

I've used a modified version of superagent-cache, it works very well to accomplish my goal of loading the site extremely fast, as it doesn't need to make any API round-trip call except for the first time. For the needs of a static website (with wordpress cms to power the api) it's very good. I get all the react goodness + insane speed + seo ability + content management system :)

However I might have a bit of a challenge ahead when I need to add pagination, I will see how it goes and keep you posted.

[mmahalwy]

Awesome! Looking forward to it @dagobertrenouf