Introduction
We've all seen those cool Snapchat and Instagram filters that usually go over a person's mouth or nose or eyes. This is made possible by machine learning and some clever image positioning. In this tutorial, we'll be using face-api.js, to detect face landmarks and cloudinary to overlay images/filters over detected landmarks. We're going to be building our application using next.js.
Codesandbox
The final project can be viewed on Codesandbox.
You can find the full source code on my Github repository.
The setup
This is majorly a javascript project. Working knowledge of javascript is required. We'll also be using React.js and a bit of Node.js. Knowledge of the two is recommended but not required. In addition, we have a machine learning(ML) aspect. For this basic tutorial, you won't need any ML or Tensors knowledge. However, if you would like to train your own models or expand the functionality, you need to be conversant with the field.
Cloudinary is a service that allows developers to store different types of media, manipulate and transform the media and also optimize its delivery.
face-api.js is a JavaScript API for face detection and face recognition in the browser implemented on top of the tensorflow.js core API.
Next.js is a react framework that allows for production-grade features such as hybrid static & server rendering, file system routing, incremental static generation, and others.
Let's start by creating a new Next.js project. This is fairly easy to do using the Next.js CLI app. Open your terminal in your desired folder and run the following command.
1npx create-next-app face-landmark-filters
This scaffolds a new project called face-landmark-filters
. You can change the name to any name you'd like. Change the directory into the new face-landmark-filters
folder and open it in your favorite code editor.
1cd face-landmark-filters
Cloudinary account and credentials
It's quite easy to get started with a free cloudinary account if you do not already have one. Fire up your browser and go to cloudinary. Create an account if you don't have one then proceed to log in. Over at the console page you'll find the credentials you need.
Open your code editor and create a new file called .local.env
at the root of your project. We're going to be putting our environment variables in this file. In case you're not familiar with environment variables, they allow us to abstract sensitive keys and secrets from our code. Read about support for environment variables in Next.js from the documentation.
Paste the following inside your .env.local
file.
1CLOUD_NAME=YOUR_CLOUD_NAME23API_KEY=YOUR_API_KEY45API_SECRET=YOUR_API_SECRET
Replace YOUR_CLOUD_NAME
, YOUR_API_KEY
and YOUR_API_SECRET
with the cloud name, api key and api secret values that we got from the cloudinary console page.
Libraries and dependencies
We're going to need a few node packages for this project.
The reason why we're using @vladmandic/face-api instead of face-api.js is because face-api.js doesn't seem to be actively maintained and is not compatible with newer versions of tensorflow.js.
@tensorflow/tfjs-node speeds up face and landmark detection using the ML models. It's not required but nice to have the speed boost.
canvas will patch the Node.js environment to have support for graphical functions. It patches the HTMLImageElement
and HTMLCanvasElement
.
formidable will be responsible for parsing any form data that we receive in our api routes.
Run the following command to install all of the above
1npm install cloudinary formidable @vladmandic/face-api @tensorflow/tfjs-node canvas
Machine Learning models
face-api.js requires some pre-trained machine learning models that will allow TensorFlow to detect faces as well as facial landmarks. As I mentioned earlier, if you'd like to train your own models or extend the functionality, you need to have knowledge of ML and deep learning. The creator of face-api.js was generous enough to provide some pre-trained models along with the library. Download the models at https://github.com/vladmandic/face-api/tree/master/model and save them in your project inside the public/models
folder. You can also get the full source code for this tutorial on my github with all the models already added.
Filter images
We also need the images that we are going to be using as filters. These need to be PNGs with a transparent background. For ease, you can download the images from https://github.com/newtonmunene99/face-landmark-filters/blob/master/public/images and save them inside the public/images
folder. Again, you can also get the full source code for this tutorial on my github with all the images already added.
Getting started
Create a folder named lib
at the root of your project. Inside this folder create a file called parse-form.js
. Paste the following code inside lib/parse-form.js
1// lib/parse-form.js2345import { IncomingForm, Files, Fields } from "formidable";6789/**1011* Parses the incoming form data.1213*1415* @param {NextApiRequest} req The incoming request object1617* @returns {Promise<{fields:Fields;files:Files;}>} The parsed form data1819*/2021export const parseForm = (req) => {2223return new Promise((resolve, reject) => {2425// Create a new incoming form2627const form = new IncomingForm({ keepExtensions: true, multiples: true });28293031form.parse(req, (error, fields, files) => {3233if (error) {3435return reject(error);3637}38394041return resolve({ fields, files });4243});4445});4647};
This file exports a function called parseForm
. This will use formidable to parse any requests that we receive in our api routes with the `multipart/form-data content-type header. Read about the specifics in the formidable docs.
Create another file inside the lib
folder and name it constants.js
. Paste the following code inside lib/constants.js
1// lib/constants.js2345/**67* @typedef {Object} Preset89* @property {number} widthOffset1011* @property {number} heightOffset1213*/14151617/**1819* @typedef {Object} Filter2021* @property {string} publicId2223* @property {string} path2425* @property {string} landmark2627* @property {Preset} presets2829*/30313233/**3435* Cloudinary folder where images will be uploaded to3637*/3839export const CLOUDINARY_FOLDER_NAME = "face-landmark-filters/";40414243/**4445* Cloudinary folder where filters will be uploaded to4647*/4849export const FILTERS_FOLDER_NAME = "filters/";50515253/**5455* Facial landmarks5657*/5859export const LANDMARKS = {6061LEFT_EYE: "left_eye",6263RIGHT_EYE: "right_eye",6465NOSE: "nose",6667MOUTH: "mouth",6869};70717273/**7475* Filters that we can apply to the image7677* @type {Filter[]}7879*/8081export const FILTERS = [8283{8485publicId: "snapchat_nose",8687path: "public/images/snapchat_nose.png",8889landmark: LANDMARKS.NOSE,9091presets: {9293widthOffset: 50,9495heightOffset: 50,9697},9899},100101{102103publicId: "clown_nose",104105path: "public/images/clown_nose.png",106107landmark: LANDMARKS.NOSE,108109presets: {110111widthOffset: 30,112113heightOffset: 30,114115},116117},118119{120121publicId: "snapchat_tongue",122123path: "public/images/tongue.png",124125landmark: LANDMARKS.MOUTH,126127presets: {128129widthOffset: 20,130131heightOffset: 50,132133},134135},136137];
These are just a few variables that we'll be using in our project. In the FILTERS
array, we have all the filters that we're going to be able to use. We define a public id for each filter, its path in the file system, the facial landmark over which we can apply the filter and a few presets that we'll use when applying the filter. Let me explain a bit more about the presets. So say for example we have a nose filter that is a bit small or large in pixel size. We need to make it a bit smaller or bigger so that it covers the person's nose perfectly so we define a width and height offset that we can use. To make it smaller you can use a negative value and make it bigger we use a positive value.
With that said, if you want to have more filters just store the filter images inside of the public/images
folder then add them to the FILTERS
array. Make sure the publicId
is unique for every filter.
Create a new file under the lib
folder and name it cloudinary.js
. Paste the following inside.
1// lib/cloudinary.js2345// Import the v2 api and rename it to cloudinary67import { v2 as cloudinary, TransformationOptions } from "cloudinary";89import { CLOUDINARY_FOLDER_NAME } from "./constants";10111213// Initialize the SDK with cloud_name, api_key, and api_secret1415cloudinary.config({1617cloud_name: process.env.CLOUD_NAME,1819api_key: process.env.API_KEY,2021api_secret: process.env.API_SECRET,2223});24252627/**2829* Get cloudinary uploads3031* @param {string} folder Folder name3233* @returns {Promise}3435*/3637export const handleGetCloudinaryUploads = (folder = CLOUDINARY_FOLDER_NAME) => {3839return cloudinary.api.resources({4041type: "upload",4243prefix: folder,4445resource_type: "image",4647});4849};50515253/**5455* @typedef {Object} Resource5657* @property {string | Buffer} file5859* @property {string} publicId6061* @property {boolean} inFolder6263* @property {string} folder6465* @property {TransformationOptions} transformation6667*6869*/70717273/**7475* Uploads an image to cloudinary and returns the upload result7677*7879* @param {Resource} resource8081*/8283export const handleCloudinaryUpload = ({8485file,8687publicId,8889transformation,9091folder = CLOUDINARY_FOLDER_NAME,9293inFolder = false,9495}) => {9697return cloudinary.uploader.upload(file, {9899// Folder to store the image in100101folder: inFolder ? folder : null,102103// Public id of image.104105public_id: publicId,106107// Type of resource108109resource_type: "auto",110111// Transformation to apply to the video112113transformation,114115});116117};118119120121/**122123* Deletes resources from cloudinary. Takes in an array of public ids124125* @param {string[]} ids126127*/128129export const handleCloudinaryDelete = (ids) => {130131return cloudinary.api.delete_resources(ids, {132133resource_type: "image",134135});136137};
This file contains all the functions we need to communicate with cloudinary. At the top, we import the v2
API from the SDK and rename it to cloudinary
for readability purposes. We also import the CLOUDINARY_FOLDER_NAME
variable from the lib/constants.js
file that we created earlier. We then proceed to initialize the SDK by calling the config
method on the api and passing to it the cloud name, api key, and api secret. Remember we defined these as environment variables in our .env.local
file earlier. The handleGetCloudinaryUploads
function calls the api.resources
method on the api to get all resources that have been uploaded to a specific folder. Read about this in the cloudinary admin api docs. handleCloudinaryUpload
calls the uploader.upload
method to upload a file to cloudinary. It takes in a resource object which contains the file we want to upload, an optional publicId, transformation object, whether or not to place the file inside a folder, and a folder name. Read more about the upload method in the cloudinary upload docs. handleCloudinaryDelete
passes an array of public IDs to the api.delete_resources
method for deletion. Read all about this method in the cloudinary admin api docs
Create a new file under the lib
folder and name it face-api.js
. Paste the following inside lib/face-api.js
.
1// lib/face-api.js23import "@tensorflow/tfjs-node";45import { Canvas, Image, ImageData, loadImage } from "canvas";67import { env, nets, detectAllFaces, Point } from "@vladmandic/face-api";8910env.monkeyPatch({ Canvas, Image, ImageData });11121314let modelsLoaded = false;15161718const loadModels = async () => {1920if (modelsLoaded) {2122return;2324}25262728await nets.ssdMobilenetv1.loadFromDisk("public/models");2930await nets.faceLandmark68Net.loadFromDisk("public/models");3132modelsLoaded = true;3334};35363738/**3940* Detect all faces in an image and their landmarks4142* @param {string} imagePath4344*/4546export const detectFaceLandmarks = async (imagePath) => {4748await loadModels();49505152const image = await loadImage(imagePath);53545556const faces = await detectAllFaces(image).withFaceLandmarks();57585960return faces;6162};63646566/**6768* Gets the approximate center of the landmark6970* @param {Point[]} landmark7172*/7374export const getCenterOfLandmark = (landmark) => {7576const coordinates = landmark.map((xy) => [xy.x, xy.y]);77787980const x = coordinates.map((xy) => xy[0]);8182const y = coordinates.map((xy) => xy[1]);83848586const centerX = (Math.min(...x) + Math.max(...x)) / 2;8788const centerY = (Math.min(...y) + Math.max(...y)) / 2;89909192return { x: centerX, y: centerY };9394};95969798/**99100* Get the approximate height and width of the landmark.101102* @param {Point[]} landmark103104* @returns105106*/107108export const getHeightWidthOfLandmark = (landmark) => {109110const minX = Math.min(...landmark.map((xy) => xy.x));111112const maxX = Math.max(...landmark.map((xy) => xy.x));113114115116const minY = Math.min(...landmark.map((xy) => xy.y));117118const maxY = Math.max(...landmark.map((xy) => xy.y));119120121122return {123124width: maxX - minX,125126height: maxY - minY,127128};129130};
This file contains all the code we need to detect faces and their landmarks. At the top we first patch the node environment so that our face-api library can be able to use the HTMLImageElement and the HTMLCanvasElement. We then have a loadModels
function which loads our pretrained models. To avoid having to load our models every time we make an API call, we have a modelsLoaded
variable that we check to see if we have already loaded the models into memory. For a normal node project, you can just load your models once when you start up your application, but since we're using Next.js and severless functions for the backend, we want to check that everytime. Read more about loading models here detectFaceLandmarks
takes in an image path, loads the ML models and then creates an Image object using the loadImage
function from the canvas package and then detects all faces with landmarks and return the faces. Read more about detecting faces and landmarks here.
getCenterOfLandmark
takes in an array of x and y coordinates then uses some simple mathematics to estimate the center of the points. Let's use an eye as an example.
12 3231 7 4455 6
Let's imagine that the numbers 1,2,3,4,5,6 above represent the outline for an eye. We want to get the center which is represented by the number 7.
getHeightWidthOfLandmark
get the approximate height and width of a landmark. It also takes in an array of x and y coordinates. Using the same example of an eye as before.
12 3231 7 4455 6
To get the approximate width, we take the smallest x coordinate which is represented by the number 1, and the largest which is represented by the number 4 then get the difference. Do the same with the height.
Let's move on to our API routes. Create a folder called filters
inside pages/api
. Create a new file called index.js
under pages/api/filters
. This file will handle calls to the /api/filters
endpoint. If you are not familiar with API routes in Next.js, I highly recommend you read the docs before proceeding. Paste the following code inside pages/api/filters/index.js
.
1// pages/api/filters/index.js2345import { NextApiHandler, NextApiRequest, NextApiResponse } from "next";67import {89handleCloudinaryUpload,1011handleGetCloudinaryUploads,1213} from "../../../lib/cloudinary";1415import {1617CLOUDINARY_FOLDER_NAME,1819FILTERS,2021FILTERS_FOLDER_NAME,2223} from "../../../lib/constants";24252627/**2829* @type {NextApiHandler}3031* @param {NextApiRequest} req3233* @param {NextApiResponse} res3435*/3637export default async function handler(req, res) {3839const { method } = req;40414243switch (method) {4445case "GET": {4647try {4849const result = await handleGetRequest();50515253return res.status(200).json({ message: "Success", result });5455} catch (error) {5657return res.status(400).json({ message: "Error", error });5859}6061}62636465default: {6667return res.status(405).json({ message: "Method not allowed" });6869}7071}7273}74757677const handleGetRequest = async () => {7879const filters = [];80818283const existingFilters = await handleGetCloudinaryUploads(8485`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}`8687);88899091filters.push(...existingFilters.resources);92939495const nonExistingFilters = FILTERS.filter((filter) => {9697const existingFilter = existingFilters.resources.find((resource) => {9899return (100101resource.public_id ===102103`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`104105);106107});108109110111return existingFilter === undefined;112113});114115116117for (const filter of nonExistingFilters) {118119const uploadResult = await handleCloudinaryUpload({120121file: filter.path,122123publicId: filter.publicId,124125inFolder: true,126127folder: `${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}`,128129});130131132133filters.push(uploadResult);134135}136137138139return filters;140141};
I am assuming that you're now already familiar with the structure of a Next.js API route. It's usually a default export function that takes in the incoming request object and the outgoing response object. In our handler function, we use a switch statement to differentiate among the different HTTP request methods. On this endpoint, api/filters
, we only want to handle GET requests.
The handleGetReqeust
function gets all filters that have been uploaded to cloudinary by calling the handleGetCloudinaryUploads
and passing in a folder. In this case, our folder will resolve to face-landmark-filters/filters/
. We then compare with the filters that we have defined in the FILTERS
array that we defined earlier inside lib/constants.js
. If the filter exists in the FILTER
array but not on cloudinary we push it into an array and then upload all filters in the array to cloudinary. We then return all filters that have been uploaded to the face-landmark-filters/filters/
folder on cloudinary.
Create a folder called images
inside pages/api
. Create a new file called index.js
under pages/api/images
. This file will handle calls to the /api/images
endpoint. Paste the following code inside pages/api/images/index.js
.
1// pages/api/images/index.js23import { NextApiHandler, NextApiRequest, NextApiResponse } from "next";45import {67handleCloudinaryUpload,89handleGetCloudinaryUploads,1011} from "../../../lib/cloudinary";1213import {1415CLOUDINARY_FOLDER_NAME,1617FILTERS,1819FILTERS_FOLDER_NAME,2021} from "../../../lib/constants";2223import {2425detectFaceLandmarks,2627getCenterOfLandmark,2829getHeightWidthOfLandmark,3031} from "../../../lib/face-api";3233import { parseForm } from "../../../lib/parse-form";34353637export const config = {3839api: {4041bodyParser: false,4243},4445};46474849/**5051* @type {NextApiHandler}5253* @param {NextApiRequest} req5455* @param {NextApiResponse} res5657*/5859export default async function handler(req, res) {6061const { method } = req;62636465switch (method) {6667case "GET": {6869try {7071const result = await handleGetRequest();72737475return res.status(200).json({ message: "Success", result });7677} catch (error) {7879console.error(error);8081return res.status(400).json({ message: "Error", error });8283}8485}86878889case "POST": {9091try {9293const result = await handlePostRequest(req);94959697return res.status(201).json({ message: "Success", result });9899} catch (error) {100101console.error(error);102103return res.status(400).json({ message: "Error", error });104105}106107}108109110111default: {112113return res.status(405).json({ message: "Method not allowed" });114115}116117}118119}120121122123const handleGetRequest = async () => {124125const result = await handleGetCloudinaryUploads();126127128129result.resources = result.resources.filter(130131(resource) =>132133!resource.public_id.startsWith(134135`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}`136137)138139);140141142143return result;144145};146147148149/**150151*152153* @param {NextApiRequest} req154155*/156157const handlePostRequest = async (req) => {158159// Get the form data using the parseForm function160161const data = await parseForm(req);162163164165const photo = data.files.photo;166167const {168169nose: noseFilter,170171mouth: mouthFilter,172173left_eye: leftEyeFilter,174175right_eye: rightEyeFilter,176177} = data.fields;178179180181const faces = await detectFaceLandmarks(photo.filepath);182183184185const transformations = [];186187188189for (const face of faces) {190191const { landmarks } = face;192193194195if (noseFilter) {196197const nose = landmarks.getNose();198199200201const centerOfNose = getCenterOfLandmark(nose);202203const heightWidthOfNose = getHeightWidthOfLandmark(nose);204205206207const filter = FILTERS.find((filter) => filter.publicId === noseFilter);208209210211if (!filter) {212213throw new Error("Filter not found");214215}216217218219transformations.push({220221overlay:222223`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`.replace(224225/\//g,226227":"228229),230231width:232233Math.round(heightWidthOfNose.width) + filter.presets?.widthOffset ??2342350,236237height:238239Math.round(heightWidthOfNose.height) + filter.presets?.heightOffset ??2402410,242243crop: "fit",244245gravity: "xy_center",246247x: Math.round(centerOfNose.x),248249y: Math.round(centerOfNose.y),250251});252253}254255if (mouthFilter) {256257const mouth = landmarks.getMouth();258259260const centerOfMouth = getCenterOfLandmark(mouth);261262const heightWidthOfMouth = getHeightWidthOfLandmark(mouth);263264265266const filter = FILTERS.find((filter) => filter.publicId === mouthFilter);267268269270if (!filter) {271272throw new Error("Filter not found");273274}275276277278transformations.push({279280overlay:281282`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`.replace(283284/\//g,285286":"287288),289290width:291292Math.round(heightWidthOfMouth.width) + filter.presets?.widthOffset ??2932940,295296height:297298Math.round(heightWidthOfMouth.height) +299300filter.presets?.heightOffset ?? 0,301302crop: "fit",303304gravity: "xy_center",305306x: Math.round(centerOfMouth.x),307308y: Math.round(centerOfMouth.y + heightWidthOfMouth.height),309310});311312}313314315316if (leftEyeFilter) {317318const leftEye = landmarks.getLeftEye();319320321322const centerOfLeftEye = getCenterOfLandmark(leftEye);323324const heightWidthOfLeftEye = getHeightWidthOfLandmark(leftEye);325326327328const filter = FILTERS.find(329330(filter) => filter.publicId === leftEyeFilter331332);333334335336if (!filter) {337338throw new Error("Filter not found");339340}341342343344transformations.push({345346overlay:347348`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`.replace(349350/\//g,351352":"353354),355356width:357358Math.round(heightWidthOfLeftEye.width) +359360filter.presets?.widthOffset ?? 0,361362height:363364Math.round(heightWidthOfLeftEye.height) +365366filter.presets?.heightOffset ?? 0,367368crop: "fit",369370gravity: "xy_center",371372x: Math.round(centerOfLeftEye.x),373374y: Math.round(centerOfLeftEye.y),375376});377378}379380381382if (rightEyeFilter) {383384const rightEye = landmarks.getRightEye();385386387388const centerOfRightEye = getCenterOfLandmark(rightEye);389390const heightWidthOfRightEye = getHeightWidthOfLandmark(rightEye);391392393394const filter = FILTERS.find(395396(filter) => filter.publicId === rightEyeFilter397398);399400401402if (!filter) {403404throw new Error("Filter not found");405406}407408409410transformations.push({411412overlay:413414`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`.replace(415416/\//g,417418":"419420),421422width:423424Math.round(heightWidthOfRightEye.width) +425426filter.presets?.widthOffset ?? 0,427428height:429430Math.round(heightWidthOfRightEye.height) +431432filter.presets?.heightOffset ?? 0,433434crop: "fit",435436gravity: "xy_center",437438x: Math.round(centerOfRightEye.x),439440y: Math.round(centerOfRightEye.y),441442});443444}445446}447448449450const uploadResult = await handleCloudinaryUpload({451452file: photo.filepath,453454transformation: transformations,455456inFolder: true,457458});459460461462return uploadResult;463464};
This endpoint is just slightly different from the api/filters
endpoint. In this one, we export a config
object at the top in addition to the default export function that handles the requests. The config
object instructs Next.js not to use the default built-in body-parser. This is because we're expecting form data and we want to parse this ourselves using formidable. Read more about custom config for api routes in the documentation.
This time around we want to handle GET and POST requests. handleGetRequest
gets all images uploaded to the face-landmark-filters/
folder. We also want to filter out any images inside the face-landmark-filters/filters/
folder because those are just our filter images.
handlePostRequest
takes in the incoming request object and passes it to the parseForm
function that we created earlier. This parses the incoming form data. From the data, we get the photo that has been uploaded
1// ...23const photo = data.files.photo;45// ...
as well as which filters to use for the nose, mouth, and eyes.
1// ...23const {45nose: noseFilter,67mouth: mouthFilter,89left_eye: leftEyeFilter,1011right_eye: rightEyeFilter,1213} = data.fields;1415// ...
We then call the detectFaceLandmarks
function and pass the uploaded photo to get all faces and landmarks.
1// ...23const faces = await detectFaceLandmarks(photo.filepath);45// ...
For every detected face, we get the landmarks using javascript object destructuring,
1// ...23for (const face of faces) {45const { landmarks } = face;67// ...
then we check the parsed form data to see if the user selected a filter to apply for either the nose, mouth, or eyes. If there's a filter for one of those landmarks we get the landmark coordinates, the center of the landmark, the height, and width of the landmark, and also check if the filter exists in our FILTERS
array. Using the nose as an example
1// ...23if (noseFilter) {45const nose = landmarks.getNose();6789const centerOfNose = getCenterOfLandmark(nose);1011const heightWidthOfNose = getHeightWidthOfLandmark(nose);12131415const filter = FILTERS.find((filter) => filter.publicId === noseFilter);1617// ...
For every filter that we need to apply, we push a transformation object to the transformations
array. Read about transformations in-depth in the cloudinary image transformations docs. To apply an overlay transformation, we need to pass the following transformation object
1// This is just an example using sample values23{45overlay: 'resource public id',67width: 100,89height: 100,1011crop: 'crop style', // which crop style to use if you need to crop1213gravity: 'gravity', // where to position the overlay relative to1415x: 100, // x coordinates relative to the gravity1617y: 100, // y coordinates relative to the gravity1819}
In our case, for the overlay, we're using the filter's folder + its publicId
value. This is why I mentioned earlier to make sure that publicId
is unique when adding filters to the FILTERS
array. For the width and height, we use the landmark's approximate height and width + their offset presets. For the crop value, we use fit
. Read about all possible values here. For gravity, we use xy_center
which is a special position. This places our overlay's center at our x and y values. Read about this here. For our x and y, we just use the center of the landmark.
For a bit more context, check out this documentation on placing layers on images.
Once we have our transformations ready, we upload the photo to cloudinary using the handleCloudinaryUpload
function and pass the transformations to the transformation
field.
Next thing, create a file called [...id].js
under the pages/api/images
folder. This file will handle api requests made to the /api/images/:id
endpoint. Paste the following code inside.
1// pages/api/images23import { NextApiRequest, NextApiResponse, NextApiHandler } from "next";45import { handleCloudinaryDelete } from "../../../lib/cloudinary";6789/**1011* @type {NextApiHandler}1213* @param {NextApiRequest} req1415* @param {NextApiResponse} res1617*/1819export default async function handler(req, res) {2021let { id } = req.query;22232425if (!id) {2627res.status(400).json({ error: "Missing id" });2829return;3031}32333435if (Array.isArray(id)) {3637id = id.join("/");3839}40414243switch (req.method) {4445case "DELETE": {4647try {4849const result = await handleDeleteRequest(id);50515253return res.status(200).json({ message: "Success", result });5455} catch (error) {5657console.error(error);5859return res.status(400).json({ message: "Error", error });6061}6263}64656667default: {6869return res.status(405).json({ message: "Method not allowed" });7071}7273}7475}76777879const handleDeleteRequest = async (id) => {8081const result = await handleCloudinaryDelete([id]);82838485return result;8687};
This endpoint only accepts DELETE requests. handleDeleteRequest
passes an images public id to handleCloudinaryDelete
and deletes the image from cloudinary. The destructured array syntax for the file name is used to match all routes that come after a dynamic route. For example to handle routes such as /api/images/:id/:anotherId/
or /api/images/:id/someAction/
instead of just /api/images/:id/
. Read this documentation to get a much better explanation.
We can finally move on to the front end. This is just some basic React.js and I won't be focusing too much on explaining what each bit does.
Add the following code inside styles/globals.css
1a:hover {23text-decoration: underline;45}678910:root {1112--color-primary: #ffee00;1314}15161718.button {1920background-color: var(--color-primary);2122border-radius: 5px;2324border: none;2526color: #000000;2728text-transform: uppercase;2930padding: 1rem;3132font-size: 1rem;3334font-weight: 700;3536cursor: pointer;3738transition: all 0.2s;3940min-width: 50px;4142}43444546.danger {4748color: #ffffff;4950background-color: #cc0000;5152}53545556.button:hover:not([disabled]) {5758filter: brightness(96%);5960box-shadow: 0px 2px 4px rgba(0, 0, 0, 0.2);6162}63646566.button:disabled {6768opacity: 0.5;6970cursor: not-allowed;7172}
These are some global styles that we're going to be using in our components.
Create a folder called components
at the root of your project and then create a file called Layout.js
inside it. Paste the following code inside components/Layout.js
1import Head from "next/head";23import Link from "next/link";4567export default function Layout({ children }) {89return (1011<div>1213<Head>1415<title>Face Landmarks Filters</title>1617<meta name="description" content="Face Landmarks Filters" />1819<link rel="icon" href="/favicon.ico" />2021</Head>22232425<nav>2627<Link href="/">2829<a>Home</a>3031</Link>3233<Link href="/images">3435<a>Images</a>3637</Link>3839</nav>40414243<main>{children}</main>4445<style jsx>{`4647nav {4849height: 100px;5051background-color: var(--color-primary);5253display: flex;5455flex-flow: row wrap;5657justify-content: center;5859align-items: center;6061gap: 10px;6263}64656667nav a {6869font-weight: bold;7071letter-spacing: 1px;7273}74757677main {7879min-height: calc(100vh- 100px);8081background-color: #f4f4f4;8283}8485`}</style>8687</div>8889);9091}
We're going to use this to wrap all of our components. This achieves some structural consistency and also avoids code duplication.
Paste the following code inside pages/index.js
.
1import { useCallback, useEffect, useState } from "react";23import Layout from "../components/Layout";45import Image from "next/image";67import { useRouter } from "next/router";89import {1011CLOUDINARY_FOLDER_NAME,1213FILTERS,1415FILTERS_FOLDER_NAME,1617} from "../lib/constants";18192021export default function Home() {2223const router = useRouter();24252627const [filters, setFilters] = useState(null);28293031/**3233* @type {[File, (file:File)=>void]}3435*/3637const [image, setImage] = useState(null);38394041/**4243* @type {[boolean, (uploading:boolean)=>void]}4445*/4647const [loading, setLoading] = useState(false);48495051/**5253* @type {[boolean, (uploading:boolean)=>void]}5455*/5657const [uploadInProgress, setUploadInProgress] = useState(false);58596061const getFilters = useCallback(async () => {6263try {6465setLoading(true);6667const response = await fetch("/api/filters", {6869method: "GET",7071});72737475const data = await response.json();76777879if (!response.ok) {8081throw data;8283}84858687setFilters(8889FILTERS.map((filter) => {9091const resource = data.result.find((result) => {9293return (9495result.public_id ===9697`${CLOUDINARY_FOLDER_NAME}${FILTERS_FOLDER_NAME}${filter.publicId}`9899);100101});102103104105return {106107...filter,108109resource,110111};112113}).filter((filter) => filter.resource)114115);116117} catch (error) {118119console.error(error);120121} finally {122123setLoading(false);124125}126127}, []);128129130131useEffect(() => {132133getFilters();134135}, [getFilters]);136137138139const handleFormSubmit = async (event) => {140141event.preventDefault();142143144145try {146147setUploadInProgress(true);148149150151const formData = new FormData(event.target);152153154155const response = await fetch("/api/images", {156157method: "POST",158159body: formData,160161});162163164165const data = await response.json();166167168169if (!response.ok) {170171throw data;172173}174175176177router.push("/images");178179} catch (error) {180181console.error(error);182183} finally {184185setUploadInProgress(false);186187}188189};190191192193return (194195<Layout>196197<div className="wrapper">198199<form onSubmit={handleFormSubmit}>200201{loading ? (202203<small>getting filters...</small>204205) : (206207<small>Ready. {filters?.length} filters available</small>208209)}210211212213{filters && (214215<div className="filters">216217{filters.map((filter) => (218219<div key={filter.resource.public_id} className="filter">220221<label htmlFor={filter.publicId}>222223<Image224225src={filter.resource.secure_url}226227alt={filter.resource.secure_url}228229layout="fill"230231></Image>232233</label>234235<input236237type="radio"238239name={filter.landmark}240241id={filter.publicId}242243value={filter.publicId}244245disabled={uploadInProgress}246247></input>248249</div>250251))}252253</div>254255)}256257258259{image && (260261<div className="preview">262263<Image264265src={URL.createObjectURL(image)}266267alt="Image preview"268269layout="fill"270271></Image>272273</div>274275)}276277<div className="form-group file">278279<label htmlFor="photo">Click to select photo</label>280281<input282283type="file"284285id="photo"286287name="photo"288289multiple={false}290291hidden292293accept=".png,.jpg,.jpeg"294295disabled={uploadInProgress}296297onInput={(event) => {298299setImage(event.target.files[0]);300301}}302303/>304305</div>306307308309<button310311className="button"312313type="submit"314315disabled={!image || uploadInProgress || !filters}316317>318319Upload320321</button>322323</form>324325</div>326327<style jsx>{`328329div.wrapper {330331height: 100vh;332333display: flex;334335flex-direction: column;336337justify-content: center;338339align-items: center;340341}342343344345div.wrapper form {346347width: 60%;348349max-width: 600px;350351min-width: 300px;352353padding: 20px;354355border-radius: 5px;356357display: flex;358359flex-direction: column;360361justify-content: start;362363align-items: center;364365gap: 20px;366367background-color: #ffffff;368369}370371372373div.wrapper form div.preview {374375position: relative;376377height: 200px;378379width: 100%;380381object-fit: cover;382383}384385386387div.wrapper form div.filters {388389width: 100%;390391height: 200px;392393display: flex;394395flex-flow: row wrap;396397justify-content: center;398399align-items: center;400401gap: 5px;402403}404405406407div.wrapper form div.filters div.filter {408409flex: 0 0 50px;410411display: flex;412413flex-flow: row-reverse nowrap;414415padding: 10px;416417border: 1px solid #cccccc;418419border-radius: 5px;420421}422423424425div.wrapper form div.filters div.filter label {426427position: relative;428429width: 100px;430431height: 100px;432433}434435436437div.wrapper form div.form-group {438439width: 100%;440441display: flex;442443flex-direction: column;444445justify-content: center;446447align-items: flec-start;448449}450451452453div.wrapper form div.form-group.file {454455background-color: #f1f1f1;456457height: 150px;458459border-radius: 5px;460461cursor: pointer;462463display: flex;464465justify-content: center;466467align-items: center;468469}470471472473div.wrapper form div.form-group label {474475font-weight: bold;476477height: 100%;478479width: 100%;480481cursor: pointer;482483display: flex;484485justify-content: center;486487align-items: center;488489}490491492493div.wrapper form div.form-group.file input {494495height: 100%;496497width: 100%;498499cursor: pointer;500501}502503504505div.wrapper form button {506507width: 100%;508509}510511`}</style>512513</Layout>514515);516517}
Notice the use of a number of React hooks. Read about the useState
hook here and the useCallback
and useEffect
hooks here. The docs have covered their uses pretty well and it's easy to understand. We use the useEffect
hook to call the memoized function getFilters
. getFilters
makes a GET request to the api/filters
endpoint to get all filters available. In the body of our component, we have a form where the user can select what filters to apply and also select a photo for upload. We use a radio button group to ensure the user doesn't select more than one filter for the same facial landmark. When the form is submitted, the handleFormSubmit
function is triggered. This function makes a POST request to the api/images
endpoint with the form data as the body. On success, we navigate to the /images
page that we'll be creating next. Read about useRouter
here.
Create a new file under pages/
called images.js
. Paste the following inside pages/images.js
.
1import { useCallback, useEffect, useState } from "react";23import Layout from "../components/Layout";45import Link from "next/link";67import Image from "next/image";891011export default function Images() {1213const [images, setImages] = useState([]);14151617const [loading, setLoading] = useState(false);18192021const getImages = useCallback(async () => {2223try {2425setLoading(true);26272829const response = await fetch("/api/images", {3031method: "GET",3233});34353637const data = await response.json();38394041if (!response.ok) {4243throw data;4445}46474849setImages(data.result.resources);5051} catch (error) {5253console.error(error);5455} finally {5657setLoading(false);5859}6061}, []);62636465useEffect(() => {6667getImages();6869}, [getImages]);70717273const handleDownloadResource = async (url) => {7475try {7677setLoading(true);78798081const response = await fetch(url, {});82838485if (response.ok) {8687const blob = await response.blob();88899091const fileUrl = URL.createObjectURL(blob);92939495const a = document.createElement("a");9697a.href = fileUrl;9899a.download = `face-landmark-filters.${url.split(".").at(-1)}`;100101document.body.appendChild(a);102103a.click();104105a.remove();106107return;108109}110111112113throw await response.json();114115} catch (error) {116117// TODO: Show error message to the user118119console.error(error);120121} finally {122123setLoading(false);124125}126127};128129130131const handleDelete = async (id) => {132133try {134135setLoading(true);136137138139const response = await fetch(`/api/images/${id}`, {140141method: "DELETE",142143});144145146147const data = await response.json();148149150151if (!response.ok) {152153throw data;154155}156157158159getImages();160161} catch (error) {162163} finally {164165setLoading(false);166167}168169};170171172173return (174175<Layout>176177{images.length > 0 ? (178179<div className="wrapper">180181<div className="images-wrapper">182183{images.map((image) => {184185return (186187<div className="image-wrapper" key={image.public_id}>188189<div className="image">190191<Image192193src={image.secure_url}194195width={image.width}196197height={image.height}198199layout="responsive"200201alt={image.secure_url}202203></Image>204205</div>206207<div className="actions">208209<button210211className="button"212213disabled={loading}214215onClick={() => {216217handleDownloadResource(image.secure_url);218219}}220221>222223Download224225</button>226227<button228229className="button danger"230231disabled={loading}232233onClick={() => {234235handleDelete(image.public_id);236237}}238239>240241Delete242243</button>244245</div>246247</div>248249);250251})}252253</div>254255</div>256257) : null}258259{!loading && images.length === 0 ? (260261<div className="no-images">262263<b>No Images Yet</b>264265<Link href="/">266267<a className="button">Upload some images</a>268269</Link>270271</div>272273) : null}274275{loading && images.length === 0 ? (276277<div className="loading">278279<b>Loading...</b>280281</div>282283) : null}284285<style jsx>{`286287div.wrapper {288289min-height: 100vh;290291background-color: #f4f4f4;292293}294295296297div.wrapper div.images-wrapper {298299display: flex;300301flex-flow: row wrap;302303gap: 10px;304305padding: 10px;306307}308309310311div.wrapper div.images-wrapper div.image-wrapper {312313flex: 0 0 400px;314315display: flex;316317flex-flow: column;318319}320321322323div.wrapper div.images-wrapper div.image-wrapper div.image {324325background-color: #ffffff;326327position: relative;328329width: 100%;330331}332333334335div.wrapper div.images-wrapper div.image-wrapper div.actions {336337background-color: #ffffff;338339padding: 10px;340341display: flex;342343flex-flow: row wrap;344345gap: 10px;346347}348349350351div.loading,352353div.no-images {354355height: 100vh;356357display: flex;358359align-items: center;360361justify-content: center;362363flex-flow: column;364365gap: 10px;366367}368369`}</style>370371</Layout>372373);374375}
This is a simple page. We call the getImages
function when the component is mounted. getImages
then makes a GET request to the /api/images
endpoint to get all uploaded images(These will be the images that already have a filter applied to them). For the body, we just show the images in a flexbox container. Each image has a download and delete button.
That's about it. I may have rushed over the UI part, however, the React.js and Next.js docs explain most of those things extremely well. You can always look anything up you might have issues with there.
The last thing we need to do is configure our Next.js project to be able to display images from cloudinary. Next.js does a lot of things under the hood to optimize the performance of your applications. One of these things is optimizing images when using the Image component from Next.js. We need to add Cloudinary's domain to our config file. Read more about this here. Add the following to next.config.js
. If you don't find the file at the root of your project you can create it yourself.
1module.exports = {23// ...45images: {67domains: ["res.cloudinary.com"],89},1011};
Our application is now ready to run.
1npm run dev
You can find the full source code on my Github. Remember, this is a simple implementation for demonstration purposes. You can always optimize a few things for use in the real world.