Integrate a Cloud-controlled Camera to Your IoT Product Builder Application

This tutorial uses the AE-Cloud2 (S5D9) board to control both the camera and two servo motors that can change the camera’s lens position.  The AE-Cloud2 board also connects to the web-based user portal of IoT Product Builder so that the end user can remotely set the position of the camera, send a request to take a picture, and receive the picture image instantly for viewing.

Prerequisite:

In this Quick Start Guide, you will:

  • Learn how to construct the hardware.
  • Flash the AE-Cloud2 board with a new binary image file.
  • Use Medium One Workflows to send a request to take a picture or change the camera’s position to the AE-Cloud2 board.
  • Use the Medium One Portal to interface with the end user and get the user’s request to take a picture or change the camera’s position. The portal also displays the photo image of the most recent picture received from the AE-Cloud2 board.

Here’s what you need to get started:

  • Renesas IoT Product Builder Kit that includes two pre-programmed AE-Cloud2 boards, pre-created Medium One sandbox account, pre-created dedicated User Portal and User Credentials provided by Medium One. (Mouser Buy Link)

  • Mini Pan-Tilt Kit with 2 Micro Servos (Mouser Buy Link)

  • A small card box (5” x 3.5” x 1”), jumper wires, double-sided tape and small screws/nuts.
  • A mini breadboard
  • Tools: a cutter, a small handy drill, and screwdriver
  • Windows PC or Mac OS (Note: This tutorial is based on Window PC.)
  • Ethernet Connectivity

STEP 1: Hardware Setup

The overall hardware setup is straightforward.

Servo Motor:

3 connection pins (Brown – GND, Red – 5V, Yellow – Input Control Pin)

Camera:

4 connection pins (Black – GND, Red – 5V, Yellow – UART TXD, White – UART RXD)

The camera operates at 5V, but the camera’s interface port operates at 3.3V from an internal 3.3V regulator on the camera module.  (This is not a guarantee.)

Do these IMPORTANT checks:

1. Connect the camera module alone to the 5V supply.

2. Measure the camera’s TXD/RXD pin voltage levels.

3. Ensure that each level does not exceed 3.3V.

If the camera’s TXD/RXD voltage levels are above 3.3V, do not plug the camera’s Grove cable into the AE-Cloud2 board directly because the high voltage will damage the AE-Cloud2 board permanently.     In this case, insert a 5V/3V converter module for the interface to protect the AE-Cloud2 board.   (See SparkFun Logic Level Converter – Bi-Directional for the part and help.)

Otherwise, if the TXD/RXD levels are 3.3V or less, a simple and direct connection can be used.  No level converters are needed.

5V Supply:

Both camera and motors need to connect to 5V supply for all operations.  The AE-Cloud2 board’s Arduino Header has the 5V supply pin.

If the board experiences power down on its own, add a large decoupling (>100uF) capacitor to the white breadboard between the 5V supply source and ground.

Cut the red wire from the camera’s grove cable and re-solder to a jumper wire with a pin header, so it can be connected to the 5V pin.

Here is the picture of an assembled setup with a small card box for this tutorial.

Here is the picture of the AE-Cloud2 hardware connections inside the box.  (If the board experiences power down on its own, add a large decoupling (>100uF) capacitor to the white breadboard between the 5V supply source and ground.)

STEP 2: Download Files

Here is the download link page.

Press the “Clone or download” green button.   Select “Download ZIP” for the zip file.

File Description:

STEP 3: Flash the AE-Cloud2 Board with the New Image File

Flash the board with the downloaded image file (s5d9_cell_ota.srec.)  Refer to HOW TO FLASH THE AE-CLOUD2 BOARD TO FACTORY DEFAULT Tutorial for the procedures.

This image file is generated from the new camera firmware with added functions to process any camera related requests to take a picture or change the camera position.   The firmware has the code to send the photo image data back to the cloud along with a  status message that indicates whether the operation is successful or not.  (For the camera firmware source code, contact Medium One.)

STEP 4: Connect the AE-Cloud2 Board to Medium One IoT Sandbox

With the new binary loaded, the AE-Cloud2 board is ready to be connected to the sandbox.

Since the pre-programmed AE-Cloud board has already been configured for Ethernet connectivity, you just plug the Ethernet cable into the socket on the board and it should connect to Medium One IoT sandbox automatically.  If there are any issues, check the steps in the AE-Cloud2 Quick Start Tutorial for help.

The net config text file should already contain the word “ethernet”.  If you run into  any network connection problem, check this file again.  Refer to the same tutorial for details.

Now it is the time to log into your sandbox account with your pre-created admin account and password.  This should be done before you turn on your device so that you can see the event messages sent by your device.  Again, refer to the AE-Cloud2 Quick Start Tutorial for details.

Once the device is connected to your pre-created IoT sandbox account, the dashboard will display all the MQTT event messages sent by your device in the “Real Time Event Stream” window.  Some messages have useful information such as connectivity type and MAC address. Make sure that you select your device correctly in the upper right corner next to the “X” button since there are two devices linked to your account.

In this example, the device login id is ”5B36…”   If you don’t see any event messages from your device in the “Real Time Events Stream” window, try to select your other device by clicking the down arrow (next to the “5B36..” text box).

STEP 5: Create Event Tag for Workflow 1

This step will create the camera photo capture event tag needed to build the workflow in the next step.  The workflow will contain your Python code that processes user’s event requests and sends the MQTT message to the device.

You will use the Medium One API Explorer to create the tags.    It is very easy to use.   Medium One API Explorer Doc Web page link shows how you can login as the admin and create the new event tags for your workflow.  Step-by-step instructions here should be sufficient to guide you through the process to create the tags and test your Python workflow.

Click “Resources” to expand the side menu.  Click “API Explorer”.

You will only need to use “Login to the API” and “Create events.”   Click on “login – Post.”

Login using your Device.

Below is a screenshot of an example email that provides the password for your device after you connect the AE-Cloud2 board to the IoT sandbox (See Step 4).   You will need to use this password to login.

Use the information in the email sent to you to fill these fields:

  • admin_id: Device’s login ID
  • admin_password: Device’s login password from email
  • api_key: the API key in the m1_config.txt file

Click “Send.”  If successful, you will get 200 as a response.

To create a new event tag.  Click on “events/<stream_name/<login_id>-POST”

  • Select “raw” for the stream_name field.
  • Select your device’s login ID
  • Type {“capture_photo”:false}

This tag’s name is “capture_photo” and its type is boolean.

Click “Send.”  If successful, you will get “200” back for the response.

On the Dashboard, your new event should appear shortly.

STEP 6: Write Workflow 1 – Request a Photo Capture

After the event tag has been created, you need to activate it so that you can use it to build your workflow.  Go to “Config” from the left-hand sidebar on your Medium One account and click “Data Streams”.

Click “raw” and then click the active checkboxes for “raw.capture_photo”. Then click “Save Data Stream”.

Now go to your Workflow Studio on your Medium One account and click “Create”.

Give your workflow a name. We named ours “Request for a Photo Capture”.

Click the “Tags and Triggers” icon, which is the first icon on the right-hand sidebar.

Click “raw” and drag the “device_id” and “capture_photo” tags into the main area of the workflow.

Double click on “raw.device_id”.  A window pops up.  Uncheck the box for Trigger.  Device ID is used to identify the client device to which the data will be sent.  It is not a trigger event.  Press Save.

The asterisk on the “raw.device_id” input block should disappear.

Now click on the “Modules” icon (the second icon in the sidebar), click on “Foundation” and drag a “Base Python” module into the workflow.

Double click on the base python block.  Click to add the 2nd input called in2.   Then press “Save”.

Connect the bottom end of the switch tag to the “in1” and “in2” circles on the Base Python module.

Double-click on the Base Python module and copy and paste the following code (follow link) into the Script section, replacing any code that is originally there.   The code is also available from the download zip file.

Now click “Save and Activate”.

Checkpoint

Power up the board.  You can open the box to check the LED status.

The firmware enters the initial setup for Ethernet.  The red LED should be on.

If the network setup is successful, all the LEDs will be off.

Next the firmware will try to connect to the IoT sandbox (both green and yellow LEDs will be turned on.)

If the setup is successful, the yellow LED will be turned off.  The green LED remains on.

Wait for one minute.  Both green and yellow LEDs should turn on.  This time the firmware will initialize the camera and detect if the camera is connected to the board.

If the setup is successful, the yellow LED will be turned off.    Only the green LED remains on.

On the dashboard, you will see that the board is connected to the sandbox successfully at this point.   You can use the dashboard instead of the LED light status to know when the connection setup is done.

Now, we will test whether your workflow is built correctly.  Use the API Explorer to create the request event to take a picture.  This time the value is set to true.  Press “Send.”

Wait for one minute.  You shall see a response from the board that it has received your request.  The response indicates that the firmware got the request and is working to make a picture shot at this point.

Also, the red LED next to the lens on the camera module is turned on.  It indicates that the camera is proceeding to take a picture shot.  When it is done, the red LED will be off.

When the picture is taken, the AE-Cloud2 board will send the image data string in readable ascii Base64 code format to the sandbox.  The data will be stored as “raw.photo”.  Don’t worry if you cannot understand the data.  Later the user portal will translate this text-based string into a viewable image.

Note: After the image data, the board also sends two more events.  The “photo_capture_success” tag will be either true or false.  The “photo_event_msg” tag has a detailed status message to indicate whether the picture is taken successfully or not.

If you don’t have a camera connected, it takes longer for the firmware to complete the setup.  (This picture shows how you can disconnect the camera from the AE-Cloud2 board.)

To send another event request for a picture,  press “Send” again.

The board is connected to the sandbox as it has done before.  However, the “photo_capture_success” tag’s value is false this time.  The “photo_event_msg” tag tells you that the camera is not connected to the board.  These two tags will be used by the user portal to help users know what happened.   Obviously, you won’t see a large image text data this time.

Now, let us move to the next step which we will build the workflow to control the position of the camera.

STEP 7: Create Two Event Tags for Workflow 2

Let us create two new event tags: the x and y positions of the camera.  These two positions are controlled by two servo motors.   Both the x and y position value range is between -4 and +4.   The value of -4 corresponds to 0 degree position of the motor while the value of +4 corresponds to 180 degree position.  Below is how the event tags can be created by API Explorer.

Type {“cam_ypos”:2}   Click “Send.”

Type {“cam_xpos”:1}   Click “Send.”

There will be two events appearing on the dashboard.

STEP 8: Write Workflow 2 – Set the Position of the Camera

The event tags have been created.   But, they need to be activated before you can use them to build your workflow.  Go to “Config” from the left-hand sidebar on your Medium One account and click “Data Streams”.

Click “raw” and then click the active checkboxes for “raw.cam_xpos” and “raw.cam_ypos.” Then click “Save Data Stream”.

Now go to your Workflow Studio on your Medium One account and click “Create.”

Give your workflow a name. We named ours “Set The Camera Position.”

Click the “Tags and Triggers” icon, which is the first icon on the right-hand sidebar.

Click “raw” and drag the “device_id”, “cam_xpos” and “cam_ypos.” tags into the main area of the workflow.

Double click on “raw.device_id”.  A window pops up.  Uncheck the box for Trigger.  Device ID is used to identify the client device to which the data is sent.  It is not a trigger event.  Press Save.

The asterisk on the “raw.deevice_id” input block should disappear.

Now click on the “Modules” icon (the second icon in the sidebar), click on “Foundation” and drag a “Base Python” module into the workflow.

Double click on the base python block.  Click to add the 2nd and 3rd input called in2 and in3 respectively.   Then press “Save”.

Connect the bottom end of the switch tag to the “in1”, “in2”, and in3 circles on the Base Python module.

Double-click on the Base Python module and copy and paste the following code (follow link) into the Script section, replacing any code that is originally there.   The code is also available from the download zip file.

Now click “Save and Activate”.

Checkpoint

Set the camera’s x position to “1.”  Type {“cam_xpos”:1}   Click “Send.”

The request event appears.   After one minute delay, the board will send an acknowledgement event back to confirm that the request has been received.   Wait a few minutes.  The x position servo motor will be turning the position of the camera as requested.  When it is complete, the board will send a status message event to indicate that the new x position is set.

Set the camera’s y position to “2.”  Type {“cam_xpos”:2}   Click “Send.”

The request event appears.   After one minute delay, the board will send an acknowledgement event back to confirm that the request has been received.   Wait a few minutes.  The y position servo motor will be turning the position of the camera as requested.  When it is complete, the board will send a status message event to indicate that the new y position is set.

The workflows are complete.  Next, the user portal (HTML/Javascript code) will be built. The portal will give the end users the friendly web-based interface as well as the ability to send the requests to the AE-Cloud2 board.

STEP 9: Add Camera Page to User Portal

Log in to your user portal as the administrator.  Click the “Admin View” in the upper right corner.

Click “Config Dashboards” from the side menu.

Click the blue “Add” button.  This action will create a new page for the camera web page.

Type “Camera” for the page name.   Right below the page name shows the URL for the camera page.   Go ahead press the blue “Save and Activate Dashboard” button.

There are two code files that you have downloaded (See Step 2).  Open “camera.html” file.  Copy the content into the HTML code space here.  Click the blue “Save and Activate Dashboard” button.

You will be prompted to save the updated code.  Click “Yes, continue.”

Click “OK.”

Place the mouse over “Javascript.”  The text will become visible.  Click on it.  The text color will change from black to green.

There are two code files that you have downloaded.  Open camera.js file.  Copy the content into the JavaScript code space here.

Click the blue “Save and Activate Dashboard” button.

You will be prompted to save the updated code.  Click “Yes, continue.”

Click “OK.”

The camera page is ready now.    But, we need to modify the default (the device’s main) page slightly.

STEP 10: Add Camera Button to Device Status Page

The device status page is the default page you will see whenever you open your device.  In order to jump from that page to your new camera page, some small changes are needed.  Before any changes are made, copy the current working code (both HTML/JavaScript) and save it somewhere on your PC.  If you make a mistake, you can restore the original code.

For HTML code, add these two lines of code that adds a camera button to the device status page.

Click the blue “Save and Activate Dashboard” button.

You will be prompted to save the updated code.  Click “Yes, continue.”

Click “OK.”

Place the mouse over “JavaScript.”  The text will become visible.  Click on it.  The text color will change from black to green.

For JavaScript code, add the following line to process the camera button.

Click the blue “Save and Activate Dashboard” button.

You will be prompted to save the updated code.  Click “Yes, continue.”

Click “OK.”

Now it is time to test your camera page.

Checkpoint

Click on the “Jump to Account” in the upper right hand corner.

A list of accounts will appear.  For your setup, you may only have one option to choose.  Select it.

On this page, you will find the two AE-Cloud2 boards that have been linked to your account.  If you had already powered up your board, the board’s status would be “online” instead of “offline.”   Click on that device.

You will see the device status page (the default page).  Click the “Camera” button.

Congratulations. Here is your camera page.   There are three buttons on the page.  They correspond to the three tags you have created with API Explorer.   Instead of using API Explorer, the camera page allows you to make the request to take a picture and change the position of the camera by pressing user-friendly buttons.

Enter “-3” for the camera’s position.   Press  the “Send Camera X Position” button.   Then, refresh the web page.

The status shows that the camera is moving to the new position.   At this point, refresh the web page until the status shows that the action is complete.

The status now shows that it is complete.

Enter “0” for the camera’s position.   Press the “Send Camera Y Position” button.   Then, refresh the web page.

The status shows that the camera is moving to the new position.   At this point, refresh the web page until the status shows that the action is complete.

Now, the status shows that it is complete.

Now, the position is set.   Press the “Photo Capture” button to take a picture.

The status shows that the camera is working on it.   At this point, refresh the web page until the status shows that the action is complete.

Congratulations!  It is done.  Your new picture is being displayed.

What’s Next?

Learn more about Medium One.  Getting Started with Medium One [Link]

Author: Rochelle Drenan