Fast JS Search on Gatsby: Roll Your Own Site Search # Fast JS Search on Gatsby: Roll Your Own Site Search #

🔥 Fast JS Search on Gatsby #

In this post, we look at how to add fast JS search on Gatsby. We are talking about site search here. That is, allowing users to search for content on your site from within your site. Of course, you could use one of the popular search services to provide the search functionality. However, if you are the kind of developer who likes to get their hands dirty and roll their own functionality, this post is for you. We will even see how you can keep your site static, while providing site search using JS Search.

JS Search was developed by Brian Vaughan from Facebook's core React team . Based on Lunr, it is so lightweight that we can build a search index just in time. Despite being lightweight, JS Search is highly configurable. We take advantage of the configurability to add stemming  so that we return similar results. As an example, if the user searches for “like”, we return results which match likes, liked, likely and liking. If all of this sounds like your cup of tea, let's get going.

📝 Plan of Action #

We will make search available to the user via the main home page on the example.com/ route. Search will be triggered when an s search parameter is included in the URL. So if the use goes to https://example.com/, the regular home page is loaded. However, if instead they go to https://example.com/?s=cheese, we will render the search component and show search results for “cheese”. Of course, the user will not need to know this to trigger a search — when the use clicks the search button, we will direct them to the search page (by navigating to https://example.com/?s= programmatically). Let’s look at how we will implement this.

1. We will use the Climate Gatsby Starter, so we can hit the ground running  and save ourselves pasting in boilerplate code. If you are adding search to an existing site, it should still be easy to follow along.
2. Our first step will be to make the search documents available to the home page component, so it can build an index when needed. JS Search is so fast that there is no real advantage to building the search index ahead of time and serializing it. We will make the documents available using Gatsby's onCreatePage API  to add the documents to the home page's context.
3. Next we will create a Search component with placeholder content and add a search button to our nav bar.
4. With all the preliminaries out of the way, we can update our home page to render the Search component instead of its regular content whenever needed.
5. Finally, we can get into the JS Search part.

Bonus #

If you have followed one of these tutorials before, you know we like to add a little polish! How about some first class user experience (UX)? Wouldn't it be great if when the user opens up a search result, the search term is highlighted in the text? On top, we will scroll the first occurrence of the search term into view.

Does that sound like a plan? Let's get to it, then!

🗳 Poll #

How do you currently implement site search on new projects?

• External service (e.g. Algolia)
• Roll my own
• No site search

Vote

Voting reveals latest results.

🧱 Fast JS Search on Gatsby #

Getting going #

First, let's clone the Gatsby Starter Climate repo. This is an MDX blog starter with some dummy content. Essentially, we just need to add site search. To clone the repo and fire it up, type these commands in the terminal:

    
gatsby new gatsby-site-search https://github.com/rodneylab/gatsby-starter-climate.git
cd gatsby-site-search
cp .env.EXAMPLE .env.development
cp .env.EXAMPLE .env.production
gatsby develop

  

We will need a few packages to add search functionality, let's add them all now:

    
npm install js-search remark stemmer strip-markdown

  

Your new MDX blog with demo content should now be running at localhost:8000/. The site is already fully functional, and you would start customizing it now with your own content if you weren't already doing this tutorial. Take a few moments to familiarize yourself with the project files and once you've seen what's what, let's carry on.

Gatsby Node #

To tinker with the onCreatePage API, we need to create a gatsby-node.js file in the project's root folder:

gatsby-node.js

javascript

    
1 const remark = require('remark');
2 const strip = require('strip-markdown');
3 const { createFilePath } = require('gatsby-source-filesystem');
4  
5 exports.onCreateNode = async ({ node, getNode, actions }) => {
6   const { createNodeField } = actions;
7   if (node.internal.type === 'Mdx') {
8     const slug = createFilePath({ node, getNode, basePath: 'content/blog' });
9     createNodeField({
10       name: 'slug',
11       node,
12       value: slug,
13     });
14   }
15 };

  

What are we doing here? Before we get on to the onCreatePage API, we need to make the post slug available, using the onCreateNode API . We will use it in onCreatePage so that later, when a user clicks a search result, they can be directed to the right post. Speaking of onCreatePage, let’s add this code to the end of gatsby-node.js:

gatsby-node.js

javascript

    
17 exports.onCreatePage = async ({ page, actions, getNodes }) => {
18   const { createPage, deletePage } = actions;
19  
20   if (page.path.match(/^/$/)) {
21     const postsRemark = await getNodes().filter((value) => value.internal.type === 'Mdx');
22     const allPosts = [];
23     postsRemark.forEach((element) => {
24       let body;
25       const { rawBody } = element;
26       const bodyStartIndex = rawBody.indexOf('---
27 ', 4);
28       const markdownBody = rawBody.slice(bodyStartIndex);
29       remark()
30         .use(strip)
31         .process(markdownBody, (err, file) => {
32           if (err) throw err;
33           body = String(file);
34         });
35       const { categories, featuredImageAlt, postTitle, seoMetaDescription, tags } =
36         element.frontmatter;
37       const { slug } = element.fields;
38       allPosts.push({
39         id: element.id,
40         body,
41         categories,
42         featuredImageAlt,
43         postTitle,
44         seoMetaDescription,
45         slug,
46         tags,
47       });
48     });
49  
50     deletePage(page);
51     createPage({
52       ...page,
53       context: {
54         ...page.context,
55         postData: {
56           allPosts,
57         },
58       },
59     });
60   }
61 };

  

onCreatePage as the name suggests, runs for every page, during the build process. We just need to add our search documents to the home page. We use a regular expression in line 20 only to run the code block for the home page. The following line (line 21) gets the site’s MDX blog posts.

One by one we will pull off data from the blog posts which we want users to be able to search for. In lines 35 – 36, you can see we are pulling data from the post front matter. The preceding code extracts the post body from its original MDX format (which includes the front matter as well as the post body itself). We place this data for each post into an array, along with an ID, which we find useful later. Finally, in lines 49 – 58, we replace the page object, adding a context. The context, like Gatsby's GraphQL API, makes data available to site pages. We will see shortly how we access this data in the page's React component.

That's all the backend stuff complete. Next, let's add some UI.

Adding a Search Component and Button #

Let's create a placeholder Search component. We will fill in content and logic later. Create a Search.jsx file in the src/components directory:

src/components/Search.jsx

jsx

    
import React from 'react';
 
const Search = () => <>Search</>;
 
export { Search as default };

  

Then we need to add a search button to the layout header, so let's edit src/components/Layout.jsx. First, we'll update imports:

src/components/Layout.jsx

jsx

    
1 import dayjs from 'dayjs';
2 import { graphql, Link, navigate } from 'gatsby';
3 import { StaticImage } from 'gatsby-plugin-image';
4 import PropTypes from 'prop-types';
5 import React from 'react';
6 import { COPYRIGHT_ENTITY } from '../constants/entities';
7 import {
8   CameraIcon,
9   FacebookIcon,
10   GithubIcon,
11   LinkedinIcon,
12   SearchIcon,
13   TwitterIcon,
14 } from './Icons';

  

Next, we want to show a search button on every page, except when the Search component is rendered. We will use a new prop, hideSearch to help:

src/components/Layout.jsx

jsx

    
119 export const PureLayout = ({ children, data: { site }, hideSearch }) => {
120   /* TRUNCATED... */
121 };

  

Then let's add the search button itself to the existing nav in the header:

src/components/Layout.jsx

jsx

    
131 <nav className={nav}>
132   <ul>
133     <li>
134       <Link aria-label="Jump to home page" href="/">
135         Home
136       </Link>
137     </li>
138     <li>
139       <Link aria-label="Jump to contact page" href="/contact/">
140         Contact
141       </Link>
142     </li>
143     {!hideSearch ? (
144       <li>
145         <button
146           type="button"
147           className={hoverJump}
148           onClick={(event) => {
149             event.preventDefault();
150             navigate('/?s=');
151           }}
152         >
153           <SearchIcon />
154         </button>
155       </li>
156     ) : null}
157   </ul>
158 </nav>

  

Here, we use Gatsby's navigate function to navigate programmatically to the search page.

To finish off in this file, let's update the prop types:

src/components/Layout.jsx

jsx

    
176 PureLayout.defaultProps = {
177   hideSearch: false,
178 };
179  
180 PureLayout.propTypes = {
181   children: PropTypes.oneOfType([PropTypes.arrayOf(PropTypes.node), PropTypes.node]).isRequired,
182   data: PropTypes.shape({
183     site: PropTypes.shape({
184       buildTime: PropTypes.string,
185       siteMetadata: PropTypes.shape({
186         facebookPage: PropTypes.string,
187         githubPage: PropTypes.string,
188         linkedinProfile: PropTypes.string,
189         twitterUsername: PropTypes.string,
190       }),
191     }),
192   }).isRequired,
193   hideSearch: PropTypes.bool,
194 };

  

This won't work yet because the SearchIcon we import doesn't yet exist. Let's create a search icon using Feather icons in the src/components/Icons.jsx file:

src/components/Layout.jsx

jsx

    
export const SearchIcon = () => (
  <span role="img" aria-label="search">
    <FeatherIcon icon="search" />
  </span>
);

  

OK, we just need to make the Search button look nice, before we move on:

src/components/Layout.module.scss

scss

    
21 .nav {
22   display: flex;
23   margin-left: auto;
24   list-style-type: none;
25  
26   ul {
27     display: flex;
28     align-items: flex-end;
29     padding-bottom: 0;
30     margin-bottom: $spacing-0;
31  
32     li {
33       display: flex;
34       font-size: $font-size-4;
35       margin-left: $spacing-6;
36       margin-bottom: $spacing-1;
37     }
38   }
39   
40   button {
41     background-color: $color-accent;
42     border-style: none;
43     color: $color-theme-4;
44     cursor: pointer;
45   }
46 }

  

If you click the search button, the URL changes, but nothing interesting happens. Let's change that in the following section.

Update Home Page with Search #

Earlier, we added a little data to our home page's context. Let’s have a look at that data now. Edit src/pages/index.jsx:

src/pages/index.jsx

scss

    
21 import { graphql } from 'gatsby';
22 import PropTypes from 'prop-types';
23 import React from 'react';
24 import BlogRoll from '../components/BlogRoll';
25 import Card from '../components/Card';
26 import { PureLayout as Layout } from '../components/Layout';
27 import Search from '../components/Search';
28 import { PureSEO as SEO } from '../components/SEO';
29 import { isBrowser } from '../utilities/utilities';
30  
31 export default function Home({ data, pageContext }) {
32   const { allPosts } = pageContext.postData;
33   if (isBrowser) {
34     const searchParam = new URLSearchParams(window.location.search.substring(1)).get('s');
35     if (searchParam !== null) {
36       return (
37         <>
38           <Layout data={data} hideSearch>
39             <Search data={data} posts={allPosts} />
40             <pre>{JSON.stringify(allPosts, null, 2)}</pre>
41           </Layout>
42         </>
43       );
44     }
45   }
46   return (
47     <>
48       <SEO
49         data={data}
50         title="Home"
51         metadescription="Climate - Gatsby v3 MDX Blog Starter - starter code by Rodney Lab to help you get going on your next blog site"
52       />
53       // TRUNCATED...
54     </>
55   )
56 };

  

Now in your browser, go to localhost:8000/?s=anything. You should see the regular home page content replaced. Instead, you will see the Search component with its placeholder content. Below that, we are rendering the context data we made available in the onCreatePage API. Isn't that awesome? If you're happy with how the context API works now, delete line 40.

A couple of notes. In line 34 we are use the URLSearchParams interface  to access the search parameters from the browser address bar. The check in line 33 just makes sure this code does not execute on the server, while the page is building, as it could break the build process (there is no browser address available during server build).

Before we move on, let's update the prop types:

src/pages/index.jsx

scss

    
54 Home.propTypes = {
55   data: PropTypes.shape({
56     site: PropTypes.shape({
57       buildTime: PropTypes.string,
58     }),
59   }).isRequired,
60   pageContext: PropTypes.shape({
61     postData: PropTypes.shape({
62       allPosts: PropTypes.arrayOf(
63         PropTypes.shape({
64           categories: PropTypes.arrayOf(PropTypes.string),
65           featuredImageAlt: PropTypes.string,
66           postTitle: PropTypes.string,
67           seoMetaDescription: PropTypes.string,
68           tags: PropTypes.arrayOf(PropTypes.string),
69         }),
70       ),
71     }),
72   }).isRequired,
73 };

  

The final step in this post is to wire up the search component. Why don't we limber up for the home straight?

JS Search #

Let's define a search options file before we crack open the Search component:

src/config/search.js

scss

    
1 const searchOptions = {
2   indexStrategy: 'Prefix match',
3   searchSanitiser: 'Lower Case',
4   indexBy: ['body', 'categories', 'featuredImageAlt', 'postTitle', 'seoMetaDescription', 'tags'],
5   termFrequency: true,
6   removeStopWords: true,
7   stemWords: true,
8 };
9   
10 export { searchOptions as default };

  

These are options we will pass to JS Search, but what do they do?

• indexStrategy: Prefix match, the default, returns a match for “cheese” with “c”, “ch”, “che” etc. Other alternatives allow substring matches from any part of the word, not just the start (so the algorithm returns a match for “cheese” with “eese”). Finally, you can request an exact match, so only “cheese” returns a match for “cheese”.
• searchSanitiser: whether we want to have case-sensitive search or not. Lower case corresponds to case-insensitive (“ChEeSe” matches “cheese”)
• indexBy: This tells JS Search which of the document data to index documents by.
• termFrequency: true switches on inverse document frequency, which basically means very common words do not artificially push results higher up the results list.
• stopWords: The algorithm ignores stop words like and, that and the  which are unlikely to be important for the search query when we set this option to true.
• stemWords: We mentioned this earlier. Going back to “cheese”, the algorithm would consider, “cheesy” a match for “cheese”.

Search Component #

With that out of the way, let's replace the Search component placeholder code:

src/config/search.js

scss

    
1 import { navigate } from 'gatsby';
2 import {
3   AllSubstringsIndexStrategy,
4   CaseSensitiveSanitizer,
5   ExactWordIndexStrategy,
6   LowerCaseSanitizer,
7   PrefixIndexStrategy,
8   Search as JSSearch,
9   StemmingTokenizer,
10   StopWordsTokenizer,
11   TfIdfSearchIndex,
12   UnorderedSearchIndex,
13 } from 'js-search';
14 import PropTypes from 'prop-types';
15 import React, { useEffect, useRef, useState } from 'react';
16 import { stemmer } from 'stemmer';
17 import searchOptions from '../../config/search';
18 import { H_ELLIPSIS_ENTITY } from '../constants/entities';
19 import { isBrowser } from '../utilities/utilities';
20 import BlogPostSummary from './BlogPostSummary';
21 import { SearchIcon } from './Icons';
22 import {
23   searchButton,
24   searchInput,
25   searchInputContainer,
26   searchLabel,
27   searchNoResultsText,
28   searchTextInput,
29 } from './Search.module.scss';
30  
31 const Search = ({ data, posts }) => {
32   const {
33     termFrequency = true,
34     removeStopWords = true,
35     stemWords = true,
36     indexStrategy = 'Prefix match',
37     searchSanitiser = 'Lower case',
38     indexBy = ['body', 'postTitle'],
39   } = searchOptions;
40  
41   let searchParam;
42   if (isBrowser) {
43     const params = new URLSearchParams(window.location.search.substring(1));
44     searchParam = params.get('s') || '';
45   }
46  
47   const [isLoading, setLoading] = useState(true);
48   const [search, setSearch] = useState(null);
49   const [searchQuery, setSearchQuery] = useState(searchParam);
50   const [searchResults, setSearchResults] = useState([]);
51   const searchInputNode = useRef();
52  
53   const addSearchIndices = (dataToSearch) => {
54     indexBy.forEach((element) => {
55       dataToSearch.addIndex(element);
56     });
57   };
58  
59   const getPostNode = (postID) => {
60     const { edges } = data.allMdx;
61     return edges.find((value) => value.node.id === postID).node;
62   };
63  
64   const getPostFrontmatter = (postData) => {
65     const { datePublished, featuredImageAlt, tags, seoMetaDescription, thumbnail, wideThumbnail } =
66       getPostNode(postData.id).frontmatter;
67     const { postTitle } = postData;
68     return {
69       datePublished,
70       featuredImageAlt,
71       thumbnail,
72       wideThumbnail,
73       postTitle,
74       seoMetaDescription,
75       tags,
76     };
77   };
78  
79   const rebuildIndex = () => {
80     const dataToSearch = new JSSearch('id');
81  
82     if (removeStopWords) {
83       dataToSearch.tokenizer = new StopWordsTokenizer(dataToSearch.tokenizer);
84     }
85     if (stemWords) {
86       dataToSearch.tokenizer = new StemmingTokenizer(stemmer, dataToSearch.tokenizer);
87     }
88     if (indexStrategy === 'All') {
89       dataToSearch.indexStrategy = new AllSubstringsIndexStrategy();
90     } else if (indexStrategy === 'Exact match') {
91       dataToSearch.indexStrategy = new ExactWordIndexStrategy();
92     } else if (indexStrategy === 'Prefix match') {
93       dataToSearch.indexStrategy = new PrefixIndexStrategy();
94     }
95  
96     dataToSearch.sanitizer =
97       searchSanitiser === 'Case sensitive'
98         ? new CaseSensitiveSanitizer()
99         : new LowerCaseSanitizer();
100     dataToSearch.searchIndex =
101       termFrequency === true ? new TfIdfSearchIndex('id') : new UnorderedSearchIndex();
102  
103     addSearchIndices(dataToSearch);
104     dataToSearch.addDocuments(posts);
105     setSearch(dataToSearch);
106     setLoading(false);
107   };
108  
109   // build the search index when the component mounts
110   useEffect(() => {
111     rebuildIndex();
112   }, []);
113  
114   // once the index is built, if we are already waiting for a search result, search and update UI
115   useEffect(() => {
116     if (searchInputNode.current) {
117       searchInputNode.current.focus();
118     }
119     if (search !== null && searchQuery !== '') {
120       const queryResult = search.search(searchQuery);
121       setSearchResults(queryResult);
122     }
123   }, [search]);
124  
125   const handleChange = (event) => {
126     const queryResult = search.search(event.target.value);
127     setSearchQuery(event.target.value);
128     setSearchResults(queryResult);
129     navigate(`A;/?s=${event.target.value}`A;);
130   };
131  
132   const handleSubmit = (event) => {
133     event.preventDefault();
134   };
135  
136   const queryResults = searchQuery === '' ? posts : searchResults;
137  
138   if (isLoading || search === null) {
139     return <p>Searching{H_ELLIPSIS_ENTITY}</p>;
140   }
141  
142   return (
143     <>
144       <form onSubmit={handleSubmit}>
145         <label htmlFor="Search">
146           <div className={searchInputContainer}>
147             <div className={searchInput}>
148             <span className={searchLabel}>Search for</span>
149               <input
150                 aria-label="Search blog posts"
151                 ref={searchInputNode}
152                 className={searchTextInput}
153                 autoComplete="off"
154                 spellCheck={false}
155                 id="Search"
156                 value={searchQuery}
157                 onChange={handleChange}
158                 placeholder="Search"
159                 type="search"
160               />
161               <button aria-labelledby="Search" type="submit" className={searchButton}>
162                 <SearchIcon />
163               </button>
164             </div>
165           </div>
166         </label>
167       </form>
168       {searchQuery === '' ? null : (
169         <>
170           {searchResults.length ? (
171             <>
172               <h1>{`A;Search results (${searchResults.length})`A;}:</h1>
173               <section role="feed">
174                 {queryResults.map((value, index) => {
175                   const { id } = value;
176                   const frontmatter = getPostFrontmatter(value);
177                   return (
178                     <article aria-posinset={index + 1} aria-setsize="-1" key={id}>
179                       <BlogPostSummary frontmatter={frontmatter} slug={value.slug.slice(1)} />
180                     </article>
181                   );
182                 })}
183               </section>
184             </>
185           ) : (
186             <p className={searchNoResultsText}>
187               Nothing like that here! Why don&rsquo;t you try another search term?
188             </p>
189           )}
190         </>
191       )}
192     </>
193   );
194 };
195  
196 Search.propTypes = {
197   data: PropTypes.shape({
198     allMdx: PropTypes.shape({
199       edges: PropTypes.arrayOf(
200         PropTypes.shape({
201           node: PropTypes.shape({
202             id: PropTypes.string.isRequired,
203             frontmatter: PropTypes.shape({
204               datePublished: PropTypes.string.isRequired,
205             }),
206           }),
207         }),
208       ),
209     }),
210   }).isRequired,
211  
212   posts: PropTypes.arrayOf(
213     PropTypes.shape({
214       catergories: PropTypes.arrayOf(PropTypes.string),
215       featuredImageAlt: PropTypes.string,
216       postTitle: PropTypes.string,
217       seoMetaDescription: PropTypes.string,
218       tags: PropTypes.arrayOf(PropTypes.string),
219     }),
220   ).isRequired,
221 };
222  
223 export { Search as default };

  

Let's look at this a bit at a time.

In lines 32 – 39, we import the options we just defined. Then in lines 41 – 45 we get the search parameter, this is similar to what we did on the home page. We need to do this, so we know what the user intends to search for. Later, we will set the URL based on the search term the user enters into the search box. The search parameter is the single source of truth, when it comes to the user's search term.

Lines 53 – 57 & 79 – 107 configure the search using the options we just set. We use React's useEffect API with an empty array as the second argument, so that the search indices are only built once, when the component mounts in lines 108 – 110. When JS Search builds the search index, it generates the search object and stores this in the component's state (line 104). Now, whenever we have a search query, we just use the search object (which has the index embedded) to get our search results.

What happens when the user types a search query in the search box? Let's look at that step, by step. With each key press, the handleChange method is called. We could add a throttle function here, but JS Search is pretty fast, so there's probably no need, unless we have many more documents. Anyway, the handleChange method (lines 125 – 130), gets the search results from our search object, sets the UI to display the search results and changes the browser URL to match the search query.

JS Search on Gatsby: Search Results #

If there are some search results, we display a BlogPostSummary component for each result. This is the same component we use on the regular home page, just instead of feeding it all posts in date order, we display search matches, most relevant first.

That was a huge chunk of code, and we only looked at the most important parts, so do let me know if some aspects need a touch more clarification.

💯 Try it out #

Please enable JavaScript to watch the video 📼

Try searching for “camera”. This is in every post, so all three dummy posts appear in results. Now if we start over, this time searching for “fold”, you see the two posts where folding cameras are mentioned appear. Click a result and the browser opens it up for you. Next search for “Elon Musk” — no results! Looks like it's all working.

🙌🏽 It’s not Right but it’s OK #

In this post, we learned:

• about JS Search configuration options;
• how we can pass data to a component's context using Gatsby's onCreatePage API; and
• using the above to add search to your Gatsby site.

We are still missing the UX polish. No need to worry, though, we will add the shine in this follow-up post on highlighting search results using Web APIs. You can see the full code for this JS Search on Gatsby on the Rodney Lab GitHub repo . You can visit the live demo site at gatsby-site-search.rodneylab.com .

Keen to hear how you plan to use this! Is there something from this post you can leverage for a side project or even client project? I do hope so! Let me know if there is anything in the post that I can improve on, for anyone else creating this project. You can leave a comment below, @ me on Twitter  or try one of the other contact methods listed below.

🙏🏽 Fast JS Search on Gatsby: Feedback #

If you have found this post useful, even though you can only afford even a tiny contribution, please consider supporting me through Buy me a Coffee.

Finally, feel free to share the post on your social media accounts for all your followers who will find it useful. As well as leaving a comment below, you can get in touch via @askRodney on Twitter and also askRodney on Telegram . Also, see further ways to get in touch with Rodney Lab. I post regularly on SvelteKit as well as Gatsby JS among other topics. Also, subscribe to the newsletter to keep up-to-date with our latest projects.