Data Fetching and Caching with React Query

I do not know whether as a techie you were enthralled by the technology used by various media houses to bring real time vote tally data. Apart, from the applications set up by media houses other developers developed similar applications but the techniques used to obtain the data was quite ingenius. They downloaded all the form 34As PDF files with Puppeteer automatically Learn More and employed Computer Vision to determine the various candidate's votes. The figures were committed to a database. The data was updated automatically in the frontend as soon as changes are made to the database.

The PDF files contained an image of the 34A form filled with the respective results of all the candidates. The figures were written down by the returning officer. Therefore, the best method of automatically obtaining the data would be to employ Machine Learning, specifically Computer Vision. So, once all the files are downloaded for all the constituencies data is preprocessed. A neural network is developed that fits the training data correctly. It is tested to see how accurately it can identify the figures written on the form. Well, before testing the neural network on the testing data you could try using MNIST dataset.

Once the Neural Network can identify correctly the figures on the form, a web application with a frontend and backend is developed. Backend consisting mainly of APIs to automatically download the PDF files when they're uploaded on IEBC's portal and feed the file to the neural network and obtain the figures for each candidate and commit it to the database. At the heart of data fetching and caching, React Query is used to ensure server changes reflect instantly in the frontend.

Huh! You are probably wondering how does one obtain data from another application. Doing a comprehensive search online, you may probably come across the following methods: fetching data from an API endpoint with free access or scraping data from the web application. An Application Programming Interface endpoint is an interface that allows two applications to talk to each other. A good illustration of what an API endpoint is in a restaurant setting where there are customers who are sitted and there are waiters attending to customer's orders. The meal is being prepared in the kitchen by chefs. In this scenario, waiters act as an API endpoint since they receive client orders and bring customers the food they ordered from the kitchen.

Web scraping is an automatic method of obtaining large amounts of data from a web application. Most of this data is unstructured and is in HTML format. This data is first converted into structured data so that it can be used.

Another question that may be lingering in your mind if you have experience managing state of components in an application is, "How will I maintain the state of your components based on the updated data? "

Managing the state of the various components in your application is very key to ensure it updates correctly based on the changes made on the server. There are various popular libraries for fetching and caching data and equally handling state:

1. React Query (renamed to TanStack Query)
2. Redux Toolkit Query (RTK)
3. Mobx State Tree
4. SWR
5. Apollo Client

1. Redux Toolkit Query

It is a powerful data fetching and caching tool. It is designed to simplify common cases for loading data in a web application, eliminating the need to hand-write data fetching and caching logic yourself.

2. React Query ( now known as TanStack Query )

Out of the box, React applications do not come with an opinionated way of fetching or updating data from your components so developers end up building their own ways of fetching data. This usually means cobbling together component-based state and effect using React hooks, or using more general purpose state management libraries to store and provide asynchronous data throughout their apps.

While most traditional state management libraries are great for working with client state, they are not so great at working with async or server state. This is because server state is totally different. For starters, server state:

• Is persisted remotely in a location you do not control or own
• Requires asynchronous APIs for fetching and updating
• Implies shared ownership and can be changed by other people without your knowledge
• Can potentially become "out of date" in your applications if you're not careful

Once you grasp the nature of server state in your application, even more challenges will arise as you go, for example:

• Caching... (possibly the hardest thing to do in programming)
• Deduping multiple requests for the same data into a single request
• Updating "out of date" data in the background
• Knowing when data is "out of date"
• Reflecting updates to data as quickly as possible
• Performance optimizations like pagination and lazy loading data
• Managing memory and garbage collection of server state
• Memoizing query results with structural sharing

React Query is hands down one of the best libraries for managing server state. It works amazingly well out-of-the-box, with zero-config, and can be customized to your liking as your application grows.

React Query allows you to defeat and overcome the tricky challenges and hurdles of server state and control your app data before it starts to control you.

React Query is the best tool I have ever stumbled upon with data refetches triggered by simple things such as a window focus or when users go to different browser tabs, and come back to your app. All of this happens without a loading spinner being shown, and your component will not re-render if the data is the same as you currently have in the cache.

In order to fetch data automatically from the server we define an asynchronous function that makes an API call to the backend endpoint.

Using useQuery hook provided out of the hood by React Query you can create a query function that will be triggerred when a window is focused or when the cached data is stale.

In order to make a POST request, for example when submitting a sign up form, we use the useMutation hook provided by React Query.

The snapshot below shows profile data of a user that is loaded to a frontend application. REST APIs were developed using Django REST.

Here is a snapshot of data displayed in the frontend:

In order to see the cached data, you can use React Query Dev Tools. Snapshot below shows cached data in React Dev Tools:

React Query is a great hook library for managing data requests that completely removes the need to put your remote data inside the global state. You just need to tell the library where you need to fetch data, and it will handle caching, background updates, and stale data without any extra code configuration.

React Query also removes the need for useState and useEffect hooks and replaces them with a few lines of React Query logic.

If you're interested to learn more checkout React Query

If you're interested in being involved in the development of an application that receives scraped data, preprocesses the data, feeds it to a neural network, commits the figures to a database and the frontend data displays the data in real time ping me through the contact page!