Cache the data

There’s one thing we need to do now before going to work on the User Interface.

We are making lots of requests to the Google Analytics API. If we don’t do nothing, we’ll quickly exceed the rate limiting quota.

Most of those requests are unnecessary because once we got the last 30 days data, or the aggregates for yesterday, they will not change until tomorrow.

What can we do, then? We can save all the data into a JSON file, and we can load the file data if it’s still recent.

Where we’ll store the data

In Glitch, the .data folder is special: data is not migrated if you remix the project, so it’s a perfect place to use as data storage.

Data storage and retrieval

Let’s create a data storage function. We’ll call that storeData(). It writes an object to the .data/data.json file, saving it as JSON.

const fs = require('fs')

const storeData = (data) => {
  try {
    fs.writeFileSync('.data/data.json', JSON.stringify(data))
  } catch (err) {
    console.error(err)
  }
}

The companion function loadData() does the opposite: it reads from the .data/data.json file.

const loadData = () => {
  try {
    const data = JSON.parse(fs.readFileSync(dataFilePath, 'utf8'))
    return data
  } catch (err) {
    console.error(err)
    return false
  }
}

Add the data invalidation workflow

The data we cache is valid for the day it was generated. When the day changes, we’ll need to refresh the data. How can we do so?

To simplify the matter, we’ll look at the file update date. If it’s been updated yesterday, or it does not exist, we’ll fetch the data again and save the file. Otherwise, if it’s been updated today we’ll load the data stored in the file.

Today’s data is always updated live, so we’ll extract that part from our codebase, and run it every time we load the data.

We introduce the wasModifiedToday() function, which checks if a file has been modified in the current day. This function uses two support functions: getFileUpdatedDate() and isToday():

//get the last update date of a file
const getFileUpdatedDate = (path) => {
  const stats = fs.statSync(path)
  return stats.mtime
}

//check if a date is "today"
const isToday = (someDate) => {
  const today = new Date()
  return someDate.getDate() == today.getDate() &&
    someDate.getMonth() == today.getMonth() &&
    someDate.getFullYear() == today.getFullYear()
}

//check if a file was modified "today"
const wasModifiedToday = (path) => {
  return isToday(getFileUpdatedDate(path))
}

Now we can use this function in this way:

const dataFilePath = '.data/data.json'

if (wasModifiedToday(dataFilePath)) {
	//load data from the file
} else {
	//fetch updated data from Google Analytics, and store it to the local file
}

Tweak how we fetch the data to separate today’s visits from the ones we can cache

Since we can cache every data except today’s visits, let’s extract them from the getData() function to getTodayData().

This is getData() now:

async function getData() {
  const list = await getPropertiesList()

  const daysAgo30 = moment().subtract(30, 'days').format('YYYY-MM-DD')
  const daysAgo60 = moment().subtract(60, 'days').format('YYYY-MM-DD')

  const getDataOfItem = async item => {
    return {
      property: item,
      today: {
        total: (await getDailyData(item.id, 'today', 'today')),
        organic: await getDailyData(item.id, 'today', 'today', true),
      },
      yesterday: {
        total: await getDailyData(item.id, 'yesterday', 'yesterday'),
        organic: await getDailyData(item.id, 'yesterday', 'yesterday', true),
      },
      monthly: {
        total: await getDailyData(item.id, '30daysAgo', 'today'),
        improvement_total: await getDailyData(item.id, daysAgo60, daysAgo30),
        organic: await getDailyData(item.id, '30daysAgo', 'today', true),
        improvement_organic: await getDailyData(item.id, daysAgo60, daysAgo30, true)
      }
    }
  }

  const result = await Promise.all(list.map(item => getDataOfItem(item)))
	console.log(result)
}

We can simply move the today property to getTodayData():

async function getTodayData() {
  const list = await getPropertiesList()

  const getDataOfItem = async item => {
    return {
      property: item,
      today: {
        total: (await getDailyData(item.id, 'today', 'today')),
        organic: await getDailyData(item.id, 'today', 'today', true),
      }
    }
  }

  return await Promise.all(list.map(item => getDataOfItem(item)))
}

and we remove it from getData() (we also add the return value instead of the console.log():

async function getData() {
  const list = await getPropertiesList()

  const daysAgo30 = moment().subtract(30, 'days').format('YYYY-MM-DD')
  const daysAgo60 = moment().subtract(60, 'days').format('YYYY-MM-DD')

  const getDataOfItem = async item => {
    return {
      property: item,
      yesterday: {
        total: await getDailyData(item.id, 'yesterday', 'yesterday'),
        organic: await getDailyData(item.id, 'yesterday', 'yesterday', true),
      },
      monthly: {
        total: await getDailyData(item.id, '30daysAgo', 'today'),
        improvement_total: await getDailyData(item.id, daysAgo60, daysAgo30),
        organic: await getDailyData(item.id, '30daysAgo', 'today', true),
        improvement_organic: await getDailyData(item.id, daysAgo60, daysAgo30, true)
      }
    }
  }

  return await Promise.all(list.map(item => getDataOfItem(item)))
}

We wrap all the calls to the specific data retrieval functions in a getAnalyticsData() function, which first checks if the file exists, using fs.existsSync(), and then checks if it was modified today.

If so, it loads the data from there. If not, it loads the data from Google Analytics, and caches it in the file:

const getAnalyticsData = async () => {
  let data = null
  if (fs.existsSync(dataFilePath) && wasModifiedToday(dataFilePath)) {
    data = loadData()
  } else {
    data = {
      aggregate: await getData()
    }
    storeData(data)
  }
  data.today = await getTodayData()
	return data
}

const data = getAnalyticsData()
data.then(data => console.log(data))

Notice we used

data.then(data => console.log(data))

because getAnalyticsData() is an async function, and it returns a promise.

At this point, in the console logs you should first have the data fetched from Google Analytics, but in subsequent reloads the data should be loaded from the file.

---

Go to the next lesson