Neural Artwork Style Transfer Using Cloudinary

Do you know there is a super-smart algorithm that can analyze your favorite artwork and recreate any other picture in the same style? Yes, it is possible. Cloudinary, in addition to its extensive capabilities, introduced an add-on that helps integrate this algorithm into mobile and web applications with just a few lines of code.

In this post, we'll build a simple app with Next.js that extracts the artistic style of an image from a predefined source images template and applies them to the content of a target image using Cloudinary's Neural Artwork Style Transfer add-on.

Here is a link to the demo CodeSandbox.

Setting Up the Project

Create a new Next.js application using the following command:

1npx create-next-app neural-artwork-style-transfer

Next, let’s add the project dependencies using the following command:

1npm install cloudinary axios

The Cloudinary Node SDK will provide easy-to-use methods to interact with the Cloudinary APIs, while axios will serve as our HTTP client for communicating with our serverless functions.

Now we can start our application on http://localhost:3000/ using the following command:

1npm start

Setting up Cloudinary

First, sign up for a free Cloudinary account if you don’t have one already. Displayed on your account’s Management Console (aka Dashboard) are important details: your cloud name, API key, etc.

Next, let’s create environment variables to hold the details of our Cloudinary account. Create a new file called .env at the root of your project and add the following to it:

1CLOUD_NAME = YOUR CLOUD NAME HERE
2    API_KEY = YOUR API API KEY
3    API_SECRET = YOUR API API SECRET

This will be used as a default when the project is set up on another system. To update your local environment, create a copy of the .env file using the following command:

1cp .env .env.local

By default, this local file resides in the .gitignore folder, mitigating the security risk of inadvertently exposing secret credentials to the public. You can update the .env.local file with your Cloudinary credentials.

When we sign up for a Cloudinary account, we don't immediately get access to the add-ons, so to access the Neural Artwork Style add-on, you need to register. Each add-on gives us a variety of plans and their associated pricing. Fortunately, most of them also have free plans, and that’s what we will be using for our application.

To register, click on the Add-ons link in the header of the management console on your Cloudinary dashboard.

You should see the Cloudinary Add-ons page with a list of all available add-ons. Select the Neural Artwork Style Transfer add-on and subscribe to the free plan, which allows 15 artworks monthly, or you can select your preferred plan.

See here for an overview of the Neural Artwork Style Transfer add-on.

Uploading and Retrieving Source Images

There are several ways to upload images to your account, but the one that we will be using is the easiest option which is the upload widget via the GUI on your dashboard. Once the upload is successful, the asset will be available for view on your dashboard; taking a closer look at the asset, you will see the public id of the image as seen below:

We also need to rename the selected images to have a consistent prefix. I added a demo prefix to my images, as seen below.

Now, let's create a serverless function that retrieves the source images from Cloudinary.

In your /pages/api folder, create a file named getImages.js and add the following to it:

1const cloudinary = require("cloudinary").v2;
2cloudinary.config({
3  cloud_name: process.env.CLOUD_NAME,
4  api_key: process.env.CLD_API_KEY,
5  api_secret: process.env.CLD_API_SECRET,
6  secure: true,
7});
8export default async function handler(req, res) {
9  try {
10    const images = await cloudinary.api.resources({
11      type: "upload",
12      prefix: "demo",
13    });
14    res.status(200).json(images.resources);
15  } catch (error) {
16    res.status(400).json(error);
17  }
18}

We imported the Cloudinary instance and set the configuration parameters using our Cloudinary credentials. Then in the handler function, we fetched the source images we uploaded and defined the type and prefix parameters.

The images are then sent to the client; once a request is made to the route, an error is sent otherwise.

The User Interface

The frontend of our application will be made up of three components — the Aside component, which will render a sidebar that displays the source images. The SelectImage component will handle selecting a target image and previewing it. Lastly, the ProcessImage component will contain a button that, when clicked, makes the call to generate the artwork and render the response image.

In the root directory, create a folder called components and add Aside.js, SelectImage.js, and ProcessImage.js files, respectively.

We'll also need some styles to give the application a nice appearance. Copy the styles in this codeSandbox link to your styles/Home.module.css file.

Add the following to your /components/Aside.js file:

1import styles from "../styles/Home.module.css";
2
3export default function Aside({ sourceImages, temp, setTemp }) {
4  return (
5    <div className={styles.sideBar}>
6      <h4>Select an image</h4>
7      <div>
8        {sourceImages &&
9          sourceImages.map((obj) => (
10            <div key={obj.public_id} onClick={() => setTemp(obj.public_id)}>
11              <img
12                src={obj.secure_url}
13                className={temp === obj.public_id ? `${styles.border}` : ""}
14                alt={obj.public_id}
15              />
16            </div>
17          ))}
18      </div>
19    </div>
20  );
21}

In this file, we defined the Aside component, and it accepts three props. The sourceImages prop will hold an array of all the source images fetched from our serverless function. The temp prop will have the public id of the source image selected by the user, and the setTemp prop will be used to set the temp state based on the selected image.

After that, we looped through the sourceImages prop and rendered an image for each. We also added an onClick event that set the value of temp to the public id of the currently clicked image, which is then later used to add an extra class name to the image conditionally.

Now add the following to your components/SelectImage.js file:

1import styles from "../styles/Home.module.css";
2
3export default function SelectImage({ targetImage, setTargetImage }) {
4  const handleInputChange = (e) => {
5    const reader = new FileReader();
6    reader.readAsDataURL(e.target.files[0]);
7    reader.onload = function (e) {
8      setTargetImage(e.target.result);
9    };
10  };
11  return (
12    <div className={styles.upload}>
13      <h4>Select an image</h4>
14      <input type="file" onChange={handleInputChange}></input>
15      <div>{targetImage && <img src={targetImage} alt="target image" />}</div>
16    </div>
17  );
18}

In the code above, we created a component called SelectImage expects as prop the target image state value and the function to set the state value. Then in the return statement, we added an input tag with an onChange event listener that triggers the handleInputChange function. This function converts the selected target image into its base64 equivalent and uses it to set the targetImage state.

Lastly, add the following to your components/ProcessImage.js file:

1import styles from "../styles/Home.module.css";
2
3export default function ProcessImage({ temp, targetImage }) {
4  return (
5    <div className={styles.process}>
6      <button className={styles.button} disabled={!temp || !targetImage}>
7        Generate artwork
8      </button>
9    </div>
10  );
11}

The ProcessImage component expects the temp and targetImage as props. It then renders a button that gets disabled if any of the props isn't provided. i.e., if the user did not select a source or target image.

So far, everything has gone well; we've created our frontend components. Now we need to connect them and test the app in the browser. Add the following to your index.js file:

1import { useEffect, useState } from "react";
2import axios from "axios";
3import styles from "../styles/Home.module.css";
4import Aside from "../components/Aside";
5import SelectImage from "../components/SelectImage";
6import ProcessImage from "../components/ProcessImage";
7
8export default function Home() {
9  const [sourceImages, setSourceImages] = useState();
10  const [targetImage, setTargetImage] = useState("");
11  const [temp, setTemp] = useState();
12
13  useEffect(() => {
14    (async () => {
15      const res = await axios.get("/api/getImages");
16      setSourceImages(res.data);
17    })();
18  }, []);
19
20  return (
21    <div className={styles.app}>
22      <h1>Cloudinary Neural Artwork Style Transfer</h1>
23      <div className={styles.container}>
24        <Aside sourceImages={sourceImages} temp={temp} setTemp={setTemp} />
25        <SelectImage
26          targetImage={targetImage}
27          setTargetImage={setTargetImage}
28        />
29        <ProcessImage temp={temp} targetImage={targetImage} />
30      </div>
31    </div>
32  );
33}

We started by importing all our components. We then created a Home component that defines three states to hold the source images, target images, and specific source image selected by the user. We then use axios to fetch the source images from our serverless function after the component’s initial render.

Next, we rendered all the components and passed the necessary state values as props. Save the file and open the browser to preview the application.

Generating the Neural Artwork

Now that we've finished building our frontend, we can go on to the most crucial part of this article — generating the Neural artwork. Create a file called upload.js in the /pages/api folder and add the following to it:

1const cloudinary = require("cloudinary").v2;
2
3cloudinary.config({
4  cloud_name: process.env.CLOUD_NAME,
5  api_key: process.env.API_KEY,
6  api_secret: process.env.API_SECRET,
7  secure: true,
8});
9
10export default async function handler(req, res) {
11  try {
12    const image = await cloudinary.uploader.upload(
13      req.body.img,
14      {
15        folder: "art-transfer",
16      },
17      async function (error, result) {
18        const response = await cloudinary.image(`${result.public_id}.jpg`, {
19          sign_url: true,
20          transformation: [
21            { height: 700, width: 700, crop: "fill" },
22            { overlay: req.body.tempId },
23            { effect: "style_transfer", flags: "layer_apply" },
24          ],
25        });
26        res.status(200).json(response);
27      }
28    );
29  } catch (error) {
30    res.status(400).json(error);
31  }
32}
33
34export const config = {
35  api: {
36    bodyParser: {
37      sizeLimit: "5mb",
38    },
39  },
40};

In the code above, we imported the Cloudinary instance and set the configuration parameters using the Cloudinary credentials. Next, we defined the handler that uploads the image string expected in the request body to a folder in our Cloudinary account called art-transfer.

The upload method accepts a callback function that runs once the upload is complete. We generate the artwork by passing the public id of the target image as an argument to the Cloudinary image transformation method. We specify the public id of the source image as an image overlay and style_transfer as the overlay effect to apply. In addition to all that, we also added a flags key and set its value to layer_apply. Click here to learn more about other style transformation preferences.

As required by Cloudinary, we also set the sign-url parameter to true to secure the transformation URL. The response is then sent to the frontend once a request is made to the serverless function. We also exported a config object to set the payload size limit to 4mb.

Preview the Generated Artwork

Update the code in your ProcessImage.js file with the following:

1import { useState } from "react";
2import axios from "axios";
3import styles from "../styles/Home.module.css";
4
5export default function ProcessImage({ temp, targetImage }) {
6  const [output, setOutput] = useState();
7  const [status, setStatus] = useState();
8
9  const handleSubmit = (e) => {
10    e.preventDefault();
11    (async function uploadImage() {
12      setStatus("Loading....");
13      try {
14        const response = await axios.post("/api/upload", {
15          img: targetImage,
16          tempId: temp,
17        });
18        const d = /'(.+)'/.exec(response.data);
19        setOutput(d[1]);
20        setStatus("");
21      } catch (error) {
22        console.log(error);
23        setStatus("Failed");
24      }
25    })();
26  };
27
28  return (
29    <div className={styles.process}>
30      <button
31        className={styles.button}
32        onClick={handleSubmit}
33        disabled={!temp || !targetImage}
34      >
35        Generate artwork
36      </button>
37      {status ? (
38        <p>{status}</p>
39      ) : (
40        <div>{output && <img src={output} alt="output-image" />}</div>
41      )}
42    </div>
43  );
44}

In the updated code, we added an event listener to the button, and it triggers the handleSubmit function. The function executes the logic to make a request to the /api/upload route while also passing the temp and targetImage as payload. The response to this request is then saved to the output state.

Now save the changes and test the application in your browser.

Find the complete project here on GitHub.

Conclusion

In this post, we built a demo application that uses the Cloudinary Neural Style Transfer add-on and provides genius neural network and style transfer capabilities.