Navigating Real-Estate Videos With Tagged Markers

Eugene Musebe

Introduction

Videos are a great way to promote products and drive customer engagement. Companies and businesses around the world have adopted video marketing. Biteable in a recent blog post mentioned that an estimated 60% of businesses use video as a marketing tool. They further noted that 94% of marketers who use videos plan to continue. Real Estate is one sector that benefits from this. Even though some realtors are now moving towards AR and VR to give demos and tours, videos are still broadly used. What can we as developers use to try and match the user experience of augmented reality? Auto tagged markers. Giving users the ability to navigate real estate videos using video tags would be a significant enhancement. While we could do this manually by visually analyzing the video and storing the tags in, say, a database, cloud-based services such as Cloudinary offer a more automated solution. Cloudinary provides a wide range of solutions, including but not limited to: programmable media, media optimization, and dynamic asset management. In this tutorial, we will be utilizing the video upload API and Cloudinary's google automatic video tagging addon.

To view the complete application demo visit the codesandbox below :

TL;DR

Here's a summary of what we'll be doing

  1. Obtain necessary credentials from Cloudinary
  2. Upload and tag a real estate video
  3. View the video on the frontend
  4. Navigate the video with auto-tagged markers

Getting Started

This tutorial assumes that you are already familiar with Javascript and React. We will also be using Node.js and Next.js (some knowledge of the two is expected but not mandatory).

Prerequisites

Installing Node.js and NPM

There are tons of tutorials on how to do this. You can check out the official Node.js website on installation and adding the path to your environment variables. You could also check out NVM, a version manager for node. If you are a power user and might be switching between node versions often, I would recommend the latter.

A code editor

You will need a code editor. Any code editor will do. Check out Visual Studio Code, which has great support for javascript and node.

Sample video

We're going to need a sample real estate video to work with. There are numerous sources for this type of video. One way would be to download a bunch of royalty-free images and then turn them into a video where each photo spans a couple of seconds. I used this approach on https://moviemakeronline.com and was able to quickly create a short video. Here's the link to the video if you'd like to reuse it.

Cloudinary account and API keys

You will need some API credentials before making requests to Cloudinary. Luckily, you can get started with a free account immediately. Head over to Cloudinary and sign in or sign up for a free account. Once done with that, head over to your console. At the top left corner of your console, you'll notice your account details. Take note of your Cloud name API Key and API Secret. We will need those later

The fun Part

Let's go ahead and initialize a new project. You can check out different installation options on the official docs.

Open up your terminal/command line and navigate to your desired Project folder. Run the following command

1npx create-next-app

The terminal will ask for your project name. Give it any sensible name. I'm going to name mine nextjs-video-tags. The command installs a few react dependencies and scaffolds our project for us. I will not get into the react/next specifics since that is beyond the scope of this tutorial.

Change the directory into your newly created project and open the folder in your code editor.

1cd nextjs-video-tags

If you're not familiar with it yet, Next js supports API routes as well as hybrid client-side static/server-rendered pages. Let's work on the backend first.

The backend

In your code editor, navigate to the pages/api/ folder and create a new file names videos.js. This page will hold our api/videos endpoint. Read more about api routes on the official docs.

API route files in next js must export a default function that takes in two parameters; an API request object and an API response object. This should all be familiar if you've created a REST API Client in any language. Let's do that.

Inside the pages/api/videos.js file, paste the following code

1export default async function videosController(req,res){
2 return res.status(200).send(`<h1>It works</h1>`);
3}

Go ahead and run your app

1npm run dev

If you open a browser and navigate to {{BASE_URL}}/api/videos, you should get the response It works. BASE_URL might be different depending on your development environment. Typically it's http://localhost:3000 for the local development environment.

Great, our API route works. Let's now implement this to handle video uploads to Cloudinary. In a real-world use case, you would upload the video from the front using something like multer, then process the video and upload it to Cloudinary. If you don't need to process the video, you could also do this on the frontend using Cloudinary's JS SDK. For the simplicity of this tutorial, we're going to make use of the video we mentioned in the Prerequisites > Sample video section. Go ahead and place this video inside your project folder. For ease, I put it inside repository/videos/house.mp4.

Before we proceed, let's install the Cloudinary npm package. Inside your terminal, run

1npm install --save cloudinary

We need to set up a module that will initialize the Cloudinary API and prepare it for use. Let's create a new folder at the root of the project and name it utils. The folder will hold utility/shared code. Inside the folder, create a new file and name it cloudinary.js

1// utils/cloudinary.js
2
3import { v2 as cloudinary } from "cloudinary";
4
5cloudinary.config({
6 cloud_name: process.env.CLOUD_NAME,
7 api_key: process.env.API_KEY,
8 api_secret: process.env.API_SECRET,
9});
10
11export default cloudinary;

Let's go over this. At the top, we import the v2 API from the Cloudinary package that we just installed. We rename the v2 API as Cloudinary for better readability. Calling the configmethod on the API will initialize it with thecloud_name api_keyandapi_secret`. Notice the use of environment variables to store the sensitive keys. We've referenced the keys as environment variables but, we have not defined them yet. Let's do that now.

At the root of your project, create a new file and name it .env.local. Inside the file, paste the following data

1CLOUD_NAME=YOUR_CLOUD_NAME
2API_KEY=YOUR_API_KEY
3API_SECRET=YOUR_API_SECRET

Replace YOUR_CLOUD_NAME YOUR_API_KEY and YOUR_API_SECRET with the appropriate values from the Prerequisites > Cloudinary account and API keys section.

We're now ready to proceed. Open up pages/api/videos.js and paste the function below inside the file.

1// pages/api/videos.js
2import cloudinary from "../../utils/cloudinary";
3
4const handlePostRequest = () => {
5 return new Promise((resolve, reject) => {
6 cloudinary.uploader.upload(
7 "repository/videos/house.mp4",
8 {
9 // Folder to store video in
10 folder: "videos/",
11 // Id that will be used to identify the video
12 public_id: "navigating-real-estate-with-auto-tagged-markers-demo",
13 // Type of resource
14 resource_type: "video",
15 // What type of categorization to be done on the video
16 categorization: "google_video_tagging",
17 // Auto tagging threshold/confidence score
18 auto_tagging: 0.6,
19 },
20 (error, result) => {
21 if (error) {
22 return reject(error);
23 }
24
25 return resolve(result);
26 }
27 );
28 });
29};

At the top, we import the Cloudinary module that initializes the API. We then have a function named handlePostRequest. The function wraps the actual upload call in a Promise then returns that promise. Inside the upload call, we pass in the video/resource that we want to upload to Cloudinary. In our case, that's a path pointing to the video file. We then pass a few options. The most important options are the categorization and auto_tagging options. I will cover both shortly. You can read more about the options you can pass from the official docs. Finally, we pass a callback function that either resolves or rejects the promise based on the result or error, respectively. Let's cover the categorization and auto_tagging options.

  1. categorization A comma-separated list of the categorization add-ons to run on the asset. In our case, we will only be using the google_video_tagging add-on, which we also need to enable. On your browser, navigate to the Cloudinary Addons Tab and enable the Google Automatic Video Tagging add-on.
  2. auto_tagging Automatically assigns tags to an asset according to detected objects or categories with a confidence score higher than the specified value. We've used 0.6 so that we can have pretty accurate results.

We now have the upload video code in place. Let's map it to our API route. Inside the same file, pages/api/videos.js, modify the videoController function to the following.

1export default async function videosController(req, res) {
2 switch (req.method) {
3 // Handle POST http methods made to /api/videos
4 case "POST":
5 try {
6 const result = await handlePostRequest(req, res);
7
8 // Resolve the request with a status code of 201(Created)
9 return res.status(201).json({
10 message: "Success",
11 result,
12 });
13 } catch (error) {
14 // Reject the request with a response of status code 400(Bad Request)
15 return res.status(400).json({
16 message: "Error",
17 error,
18 });
19 }
20 // Reject all other http methods made to /api/videos
21 default:
22 return res.status(405).json({ message: "Method Not Allowed" });
23 }
24}

What we're doing here is setting up the route to accept POST requests and handle them. All other HTTP methods will fail with status 405. You could go ahead and accept all methods, but we're going for a little bit more control in this case. We're now ready to use the /api/videos endpoint. You can use this endpoint from any API client but let's create a simple UI for it.

The frontend

Let's create a simple page where we can upload and view the video. Open up pages/index.js in your code editor and export a react component. Let's call it Home.

1import Head from "next/head";
2import { useState, useRef, MutableRefObject } from "react";
3
4export default function Home() {
5 /**
6 * @type {MutableRefObject<HTMLVideoElement>}
7 */
8 const playerRef = useRef(null);
9
10 /**
11 * @type {[Array<UploadVideoResult>, Function]} video
12 */
13 const [videos, setVideos] = useState([]);
14
15 const [loading, setLoading] = useState(false);
16
17 const [error, setError] = useState(null);
18
19 return (
20 <div
21 style={{ width: "100vw", minHeight: "100vh", backgroundColor: "white" }}
22 >
23 <Head></Head>
24 <main
25 style={{
26 maxWidth: "1000px",
27 margin: "auto",
28 display: "flex",
29 flexFlow: "column nowrap",
30 justifyContent: "center",
31 alignItems: "start",
32 }}
33 >
34 <h1
35 style={{
36 margin: "100px auto",
37 fontWeight: "bold",
38 fontSize: "50px",
39 }}
40 >
41 Real Estate
42 </h1>
43
44 <div
45 style={{
46 height: "2px",
47 width: "100%",
48 backgroundColor: "black",
49 }}
50 ></div>
51 </main>
52 </div>
53 );
54}

We've just created a simple React Functional Component and set up a few state and ref hooks. Read about react hooks in the official docs. The first hooks will hold our videos state; these are the videos that we upload. The second hook will hold our loading state; this is to show where an upload is in progress or not. And finally, our error hook holds any errors that might arise from the upload. Apart from the state hooks, we also have a ref hook that stores a dom reference to our video player.

You might have also noticed the weird comments. These are JSDoc type definitions. JSDoc Typedefs allow us to define interfaces and types without writing our app in Typescript and hence leverage the power of intelli-sense. Let's write a few of them.

Create typedefs.js inside the utils folder and paste the following inside.

1/**
2 * @typedef {Object} UploadVideoResult
3 * @property {number} duration
4 * @property {string} format
5 * @property {number} height
6 * @property {number} width
7 * @property {string} url
8 * @property {string} secure_url
9 * @property {string} public_id
10 * @property {Info} info
11 */
12
13/**
14 * @typedef {Object} Info
15 * @property {Categorization} categorization
16 */
17
18/**
19 * @typedef {Object} Categorization
20 * @property {Category} google_video_tagging
21 */
22
23/**
24 * @typedef {Object} Category
25 * @property {Array<VideoTag>} data
26 */
27
28/**
29 * @typedef {Object} VideoTag
30 * @property {string} tag
31 * @property {Array<string>} categories
32 * @property {number} start_time_offset
33 * @property {number} end_time_offset
34 * @property {number} confidence
35 */
36
37/**
38 * @typedef {Object} TagWithRooms
39 * @property {string} tag
40 * @property {Array<VideoTag>} rooms
41 *
42 */

Go back to pages/index.js and import the file at the top.

1import "../utils/typedef";

We can now define our upload handler. Again, in a real-world scenario, you would want to have a file input and upload that to the backend or directly to Cloudinary, however, for this tutorial we're just going to be making a call to the api/videos endpoint to mock this behavior. Let's add a handler method just below our state hooks.

1const handleUploadVideo = async () => {
2 try {
3 // Set loading to true
4 setLoading(true);
5 // Clear any existing errors
6 setError(null);
7
8 // Make a POST request to the `api/videos/` endpoint
9 const response = await fetch("/api/videos", {
10 method: "post",
11 });
12
13 // Set loading to true once a response is available
14 setLoading(false);
15
16 // Check if the response is successful
17 if (response.status >= 200 && response.status < 300) {
18 const data = await response.json();
19
20 /**
21 * @type {UploadVideoResult}
22 */
23 const result = data.result;
24
25 // Update our videos state with the results
26 setVideos([...videos, result]);
27 } else {
28 throw await response.json();
29 }
30 } catch (error) {
31 // Set error state
32 setError(error);
33 console.error({ ...error });
34 }
35};

This method calls the api/videos endpoint and updates our videos state with the result. You could show an alert to the users when they upload completes or fails. Check out the official docs to understand the schema of the result object. Here's what it might look like

1{
2 public_id: 'cr4mxeqx5zb8rlakpfkg',
3 version: 1571218330,
4 signature: '63bfbca643baa9c86b7d2921d776628ac83a1b6e',
5 width: 864,
6 height: 576,
7 format: 'jpg',
8 resource_type: 'image',
9 created_at: '2017-06-26T19:46:03Z',
10 bytes: 120253,
11 type: 'upload',
12 url: 'http://res.cloudinary.com/demo/image/upload/v1571218330/cr4mxeqx5zb8rlakpfkg.jpg',
13 secure_url: 'https://res.cloudinary.com/demo/image/upload/v1571218330/cr4mxeqx5zb8rlakpfkg.jpg'
14 info:{
15 categorization:{
16 google_video_tagging:{
17 data:[
18 {
19 tag:"",
20 categories:[""],
21 start_time_offset:0,
22 end_time_offset:0,
23 }
24 ]
25 }
26 }
27 }
28}

Take special note of format, url, secure_url, and info.categorization.google_video_tagging.data. Inside info.categorization.google_video_tagging.data take note of tag,categories and start_time_offset. We're going to need these fields.

And now, let's update our UI. Modify the Home component to return the following

1return (
2 <div
3 style={{ width: "100vw", minHeight: "100vh", backgroundColor: "white" }}
4 >
5 <Head></Head>
6 <main
7 style={{
8 maxWidth: "1000px",
9 margin: "auto",
10 display: "flex",
11 flexFlow: "column nowrap",
12 justifyContent: "center",
13 alignItems: "start",
14 }}
15 >
16 <h1
17 style={{
18 margin: "100px auto",
19 fontWeight: "bold",
20 fontSize: "50px",
21 }}
22 >
23 Real Estate
24 </h1>
25
26 <div
27 style={{
28 height: "2px",
29 width: "100%",
30 backgroundColor: "black",
31 }}
32 ></div>
33
34 <div
35 style={{
36 width: "100%",
37 display: "flex",
38 flexFlow: "column nowrap",
39 justifyContent: "center",
40 alignItems: "center",
41 }}
42 >
43 <div
44 style={{
45 display: "flex",
46 flexFlow: "column nowrap",
47 justifyContent: "center",
48 alignItems: "center",
49 }}
50 >
51 <p>
52 {!videos.length ? "No Video Yet. " : ""}Tap on the button below to
53 add a video.
54 </p>
55 <button
56 style={{
57 height: "50px",
58 padding: "0 30px",
59 fontWeight: "bold",
60 }}
61 disabled={loading}
62 onClick={handleUploadVideo}
63 >
64 UPLOAD VIDEO
65 </button>
66 </div>
67 {loading || error ? (
68 <div
69 style={{
70 width: "100%",
71 display: "flex",
72 flexFlow: "column nowrap",
73 justifyContent: "center",
74 alignItems: "center",
75 }}
76 >
77 <hr style={{ width: "60%" }}></hr>
78 {loading ? (
79 <p>Please be patient while the video uploads...</p>
80 ) : null}
81 {error ? (
82 <p style={{ color: "red" }}>There was a problem</p>
83 ) : null}
84 </div>
85 ) : null}
86 {videos.length ? (
87 videos.map((video) => (
88 <div id="video-component">
89 Show video here
90 </div>
91 ))
92 ) : (
93 <div
94 style={{
95 margin: "20px auto",
96 width: "100%",
97 display: "flex",
98 flexFlow: "column nowrap",
99 justifyContent: "center",
100 alignItems: "center",
101 }}
102 >
103 <hr style={{ width: "60%" }}></hr>No videos yet
104 </div>
105 )}
106 </div>
107 </main>
108 </div>
109);

That's a lot of code. Let us go over what's happening. We have a div at the top that has an upload button. The button is disabled if loading is true. Below that, we have the loading indicator div. This div will only show when loading is true or if there's an error. After these two, we have a div that will now hold our video elements/components. We map through our uploaded videos and return a div for each video.

We're going to have a flex container with the video on the left and the navigation markers on the right. Replace the div with an id of video-component with the following code

1<div
2 id="video-component"
3 style={{
4 height: "520px",
5 display: "flex",
6 flexFlow: "row nowrap",
7 margin: "20px auto",
8 backgroundColor: "white",
9 border: "solid 2px black",
10 borderRadius: "5px",
11 }}
12>
13 <video
14 ref={playerRef}
15 controls
16 style={{
17 height: "100%",
18 width: "75%",
19 objectFit: "cover",
20 }}
21 >
22 <source src={video.secure_url} type={`video/${video.format}`} />
23 Your browser does not support the video tag.
24 </video>
25 <div
26 style={{
27 height: "100%",
28 width: "25%",
29 backgroundColor: "white",
30 padding: "8px",
31 overflowY: "auto",
32 }}
33 >
34 <h1 style={{ fontSize: "18px" }}>Location Markers</h1>
35 <hr></hr>
36 <ul style={{ listStyleType: "none", overflowY: "none" }}>
37 {video.info.categorization.google_video_tagging.data
38 .filter(
39 (tag) => tag.categories.includes("room") || tag.tag.includes("room")
40 )
41 .reduce((tags, tag) => {
42 /**
43 * @type {TagWithRooms}
44 */
45 const existingTagWithRoooms = tags?.find(
46 (tagWithRoooms) =>
47 tagWithRoooms.tag.toLowerCase() === tag.tag.toLowerCase()
48 );
49
50 if (!existingTagWithRoooms) {
51 return [
52 ...tags,
53 {
54 tag: tag.tag,
55 rooms: [tag],
56 },
57 ];
58 }
59
60 existingTagWithRoooms.rooms.push(tag);
61
62 return tags;
63 }, [])
64 .map(
65 /**
66 * @param {TagWithRooms} tag
67 */
68 (tag) => (
69 <li style={{ margin: "5px 0" }}>
70 {tag.rooms.length > 1 ? (
71 [
72 <h2 style={{ fontSize: "16px" }}>{tag.tag}</h2>,
73 <hr></hr>,
74 <ul style={{ listStyleType: "none" }}>
75 {tag.rooms.map((room, index) => (
76 <li style={{ margin: "5px 0" }}>
77 <button
78 onClick={() => {
79 playerRef.current.currentTime =
80 room.start_time_offset;
81 }}
82 >
83 {room.tag} {index + 1}
84 </button>
85 </li>
86 ))}
87 </ul>,
88 ]
89 ) : (
90 <button
91 onClick={() => {
92 playerRef.current.currentTime =
93 tag.rooms[0].start_time_offset;
94 }}
95 >
96 {tag.rooms[0].tag}
97 </button>
98 )}
99 </li>
100 )
101 )}
102 </ul>
103 </div>
104</div>

The video element is self-explanatory. We just have a native HTML Video element. We then store a reference to the element in our playerRef ref hook. We set the source of the video to the secure_url field of the upload result and the format to the format field of the upload result.

On the right side, we have an unordered list. This list will hold our navigation markers which are currently stored in the video.info.categorization.google_video_tagging.data field of the upload result. Since Cloudinary returns all sorts of tags that we don't need, we're going to filter tags that include the word room in either the tag or categories field.

1video.info.categorization.google_video_tagging.data
2.filter(
3 (tag) =>
4 tag.categories.includes("room") ||
5 tag.tag.includes("room")
6)

Next, we want to show a list item for every room and a list if there is more than one room with the same tag. i.e

1<ul>
2 <li>
3 <button>Kitchen</button>
4 </li>
5 <li>
6 <button>Bathroom</button>
7 </li>
8 <li>
9 <h2>Bedroom</h2>
10 <ul>
11 <li>
12 <button>Bedroom 1</button>
13 </li>
14 <li>
15 <button>Bedroom 2</button>
16 </li>
17 </ul>
18 </li>
19</ul>

Let's modify our data to help us achieve this. We will use Javascript's Array.reduce method to achieve the following schema. You could also use something like lodash.

1{
2 tag:"",
3 rooms:[
4 {
5 tag:"",
6 categories:[""],
7 start_time_offset:0,
8 end_time_offset:0,
9 }
10 ]
11}

Let's see how we'll do this. Chain our filter method with the following reduce method.

1.reduce((tags, tag) => {
2/**
3 * Check if a room with a similar tag has already been found
4 * @type {TagWithRooms}
5 */
6const existingTagWithRoooms = tags?.find(
7 (tagWithRoooms) =>
8 tagWithRoooms.tag.toLowerCase() ===
9 tag.tag.toLowerCase()
10);
11
12if (!existingTagWithRoooms) {
13 return [
14 ...tags,
15 {
16 tag: tag.tag,
17 rooms: [tag],
18 },
19 ];
20}
21
22existingTagWithRoooms.rooms.push(tag);
23
24return tags;
25}, [])

Finally, we map this to a list item inside our unordered list.

Navigating the video

We now have our video on the left and our tags in an unordered list on the right. Each list item wraps a button with an onclick method. This is where we will navigate the video. Remember the tag.start_time_offset field, this field tells us at what time in the video the room is shown. We're going to use this to seek the video player. When a user clicks on a button for a certain room we will set the video's currentTime to the start_time_offset

1playerRef.current.currentTime =room.start_time_offset;

You can go even further and add navigation markers to the video's progress bar using video.js and video.js marker plugin, but I won't get into that.

Congratulations~ You made it to the end of the tutorial. You can find the full codesandbox link here and the Github codebase here

Resources

You may find these resources useful.

Eugene Musebe

Software Developer

I’m a full-stack software developer, content creator, and tech community builder based in Nairobi, Kenya. I am addicted to learning new technologies and loves working with like-minded people.