PyCozmo

Overview

https://github.com/zayfod/pycozmo

PyCozmo is a pure-Python communication library, alternative SDK, and application for the Cozmo robot . It allows controlling a Cozmo robot directly, without having to go through a mobile device, running the Cozmo app.

The library is loosely based on the Anki Cozmo Python SDK and the cozmoclad (“C-Like Abstract Data”) library.

This project is a tool for exploring the hardware and software of the Digital Dream Labs (originally Anki) Cozmo robot. It is unstable and heavily under development.

Usage

Basic:

import time
import pycozmo

with pycozmo.connect() as cli:
    cli.set_head_angle(angle=0.6)
    time.sleep(1)

Advanced:

import pycozmo

cli = pycozmo.Client()
cli.start()
cli.connect()
cli.wait_for_robot()

cli.drive_wheels(lwheel_speed=50.0, rwheel_speed=50.0, duration=2.0)

cli.disconnect()
cli.stop()

PyCozmo vs. the Cozmo SDK

A Cozmo SDK application (aka “game”) acts as a client to the Cozmo app (aka “engine”) that runs on a mobile device. The low-level communication happens over USB and is handled by the cozmoclad library.

In contrast, an application using PyCozmo basically replaces the Cozmo app and acts as the “engine”. PyCozmo handles the low-level UDP communication with Cozmo.

+------------------+                   +------------------+                   +------------------+
|     SDK app      |     Cozmo SDK     |    Cozmo app     |       Cozmo       |      Cozmo       |
|      "game"      |     cozmoclad     |     "engine"     |      protocol     |     "robot"      |
|                  | ----------------> |   Wi-Fi client   | ----------------> |     Wi-Fi AP     |
|                  |        USB        |    UDP client    |     UDP/Wi-Fi     |    UDP Server    |
+------------------+                   +------------------+                   +------------------+

Requirements

Installation

Using pip:

pip install --user pycozmo

pycozmo_resources.py download

From source:

git clone https://github.com/zayfod/pycozmo.git
cd pycozmo
python setup.py install --user

pycozmo_resources.py download

From source, for development:

git clone git@github.com:zayfod/pycozmo.git
cd pycozmo
python setup.py develop --user
pip install --user -r requirements-dev.txt

pycozmo_resources.py download

Support

Bug reports and changes should be sent via GitHub:

https://github.com/zayfod/pycozmo

DDL Robot Discord server, channel #development-cozmo:

https://discord.gg/ew92haS

Disclaimer

This project is not affiliated with Digital Dream Labs or Anki.

PyCozmo Architecture

Overview

PyCozmo is designed as a multithreaded library.

It is organized in three layers with each higher layer building on the ones below it:

  • low-level connection layer
  • client or SDK layer
  • application layer

Each layer provides it’s own API and can be used independently.

The following diagram illustrates the library architecture.

                                                                             ^
                                                                             |
                    +-----------------------------------------+              |
                    |                                         |              |
                    |                  Brain                  |              |
                    |                                         |              |
                    |                                         |              |
                    +-----+--------------+--------------+-----+              |
                          ^              |              ^                    |
                          |              |              |                    |
     +-----------+  +-----+-----+        |              |                    |  Application
     |           |  |           |        |              |                    |  Layer
     | Heartbeat |  |  Reaction |        |              |                    |
     |  Thread   |  |   Thread  |        |              |                    |
     |           |  |           |        |              |                    |
     +-----+-----+  +-----+-----+        |              |                    |
           |              ^              |  Commands    |  Events            |
           |              |              |              |                    |
           |             +-+             |              |                    |
           |             |-|  Event      |              |                    |
           |             |-|  Queue      |              |                    |
           |             +-+             |              |                    |
           |              ^              |              |                    |
           |              |              |              |                    v
           +------------->+  Reactions   |              |
                          |              v              |                    ^
                    +-----+--------------+--------------+-----+              |
                    |                                         |              |
                    |                  Client                 |              |
                    |                                         |              |
                    |                                         |              |
                    +-----+--------------+--------------+-----+              |
                          |              |              ^                    |  SDK
                          v              |              |                    |  Layer
                         +-+             |              |                    |
              Animation  |-|             |  Commands    |  Events            |
                Queue    |-|             |              |                    |
                         +-+             |              |                    |
                          |              |              |                    |
                          v              v              |                    |
+-----------+       +-----+-----+  +-----+--------------+-----+              |
|           |       |           |  |                          |              |
|   Face    |       | Animation |  |        Connection        |              v
| Generator +------>+  Thread   |  |          Thread          |
|           |       |           |  |                          |              ^
+-----------+       +-----+-----+  +-----+--------------+-----+              |
                          |              |              ^                    |
                          +------------->+              |                    |
                                         |              |                    |
                                         v              |                    |
                                        +-+            +-+                   |
                              Outgoing  |-|            |-|  Incoming         |
                               Message  |-|            |-|   Message         |
                                Queue   +-+            +-+    Queue          |
                                         |              ^                    |
                                         v              |                    |  Connection
                                   +-----+-----+  +-----+-----+              |  Layer
                                   |           |  |           |              |
                                   |   Send    |  |  Receive  |              |
                                   |  Thread   |  |  Thread   |              |
                                   |           |  |           |              |
                                   +---------+-+  +--+--------+              |
                                             |       ^                       |
                                             |       |                       |
                                             |       |                       |
                                             v       |                       |
                                           +-+-------+-+                     |
                                           |           |                     |
                                           |    UDP    |                     |
                                           |   Socket  |                     |
                                           |           |                     |
                                           +-----------+                     |
                                                                             v

Connection Layer

The connection layer implements the Cozmo communication protocol.

The receive thread reads Cozmo protocol frames, encapsulated in UDP datagrams, from the UDP socket. It maintains a receive window for incoming packets and sends a stream of incoming packets in the correct order over the incoming message queue to the connection thread.

The send thread reads a stream of outgoing packets from the outgoing message queue, builds Cozmo protocol frames and sends them over the UDP socket. It maintains a send window and resends packets that are not acknowledged in time.

The connection thread reads a stream of incoming packets from the incoming message queue and dispatches them to registered handler functions. It sends ping packets on a regular basis to maintain connection with the robot.

Client Layer (SDK)

The client layer provides access to robot on-board functions.

It allows sending commands and registering handler function for incoming packets and events.

It performs:

  • camera image reconstruction
  • display image encoding
  • audio encoding
  • animation and audio playback
  • procedural face generation

The animation controller synchronizes animations, audio playback, and image display. It works as a separate thread that aims to send images and audio to the robot at 30 frames per second. All on-board function of the robot are synchronized to this framerate, including images, audio playback, backpack and cube LED animations.

Application Layer

The application layer implements high-level off-board functions:

  • reactions and behaviors
  • personality engine
  • computer vision (CV) camera image processing

Events from the client layer are converted to reactions. The reaction thread reads events from its incoming event queue and handles them appropriately. Reactions normally trigger behaviors.

The heartbeat thread drives the personality engine and timers for activities and behaviors.

Cozmo Protocol

Overview

The Cozmo protocol is a UDP-based variant of the selective repeat automatic-repeat request (ARQ) protocol.

The Cozmo app (aka “engine”) acts as a client and Cozmo (aka “robot”) acts as a server.

The two exchange frames, encapsulated in UDP packets.

Each frame can contain 0, 1, or more packets.

See protocol_declaration.py for packet details.

Network Setup

The robot acts as a Wi-Fi access point. It always uses an SSID that follows the form “Cozmo_XXXXXX”, where XXXXXX are upper-case hexadecimal digits. It acts as a DHCP server and assigns Wi-Fi clients an IP address in the range 172.31.1.0/24 .

The app searches for robot APs. If it finds only one, it will associate with it automatically. If it finds more than one, it will allow the user to select one manually.

The robot acts as a server. It always uses the IP address 172.31.1.1 and will expect UDP packets on port 5551. It will only accept packets originating from an IP address in the range 172.31.1.0/24 .

The app acts as a client and initiates connections. It will only accept packets originating from the IP address 172.31.1.1 .

+--------------------+                        +--------------------+
|     Cozmo app      |                        |       Cozmo        |
|      "engine"      |        UDP/Wi-Fi       |      "robot"       |
|    Wi-Fi client    | ---------------------> |      Wi-Fi AP      |
|     UDP client     |                        |     UDP Server     |
+--------------------+                        +--------------------+
    172.31.1.0/24                                 172.31.1.1:5551

Frames

Each frame has the following structure:

Field           Length      Description
---------------------------------------------------------------------------------
id              7           Always "COZ\x03RE\x01"
type            1           Frame type
first_seq       2           First packet sequence number in the frame or 0
seq             2           Last packet sequence number in the frame or 0
ack             2           Peer packet sequence number acknowledgement
packets         -           0 or more encapsulated packets

Frame types:

Type            Source      Description
---------------------------------------------------------------------------------
0x01            engine      Reset
0x02            robot       Reset ACK
0x03            engine      Disconnect
0x04            engine      Engine packet - single
0x07            engine      Engine packets - zero or more
0x09            robot       Robot packets - zero or more
0x0b            engine      Out-of-band engine ping

Packets

Packet types:

Type    OOB     Source      Description
---------------------------------------------------------------------------------
0x02    n       robot       Connect
0x03    n       engine      Disconnect
0x04    n       both        Command
0x05    y       robot       Event
0x0a    y       engine      Keyframe
0x0b    y       engine      Ping

Out of band packets do not get assigned sequence IDs.

Packet content is Cozmo firmware version specific.

Commands and events are identified by an 8-bit ID. IDs in the range 0-0xaf are sent by the engine. IDs in the range 0xb0-0xff are sent by the robot.

IDs in the range 0xf0-0xff are used for out-of-band updates. These are packets that are not tracked by a sequence ID and thus not retransmitted. Only their latest received value is considered important.

ID                 Min     Max      Name
---------------------------------------------------------------------------------
0x03	     	    31	    31		LightStateCenter              
0x04	     	    40	    40		CubeLights                    
0x05	     	     5	     5		ObjectConnect                 
0x0b	     	     1	     1		SetHeadLight                  
0x0c	     	     1	     1		                              
0x10	     	     5	     5		CubeId                        
0x11	     	    21	    21		LightStateSide                
0x25	     	     0	     0		Enable                        
0x32	     	    16	    16		DriveWheels                   
0x33	     	    10	    10		TurnInPlaceAtSpeed                   
0x34	     	     4	     4		DriveLift                     
0x35	     	     4	     4		DriveHead                     
0x36	     	    17	    17		SetLiftHeight                 
0x37	     	    17	    17		SetHeadAngle                  
0x39	     	    20	    20		TurnInPlace                 
0x3b	     	     0	     0		StopAllMotors
0x3d								DriveStraight             
0x45	     	    24	    24		                              
0x4b	     	     8	     8		EnableBodyACC		                              
0x4c	     	     2	     2		EnableCamera                  
0x50	     	     2	     2		                              
0x54	     	     2	     2		                              
0x57	     	     7	     7		SetCameraParams               
0x60	     	     1	     1		EnableStopOnCliff             
0x64	     	     2	     2		SetRobotVolume                
0x66	     	     1	     1		EnableColorImages             
0x80	     	     4	     4		                              
0x81	     	    12	   144	*	NvStorageOp                   
0x8d	     	     0	     0		                              
0x8e	     	   744	   744		OutputAudio                   
0x8f	     	     0	     0		OutputSilence                     
0x93	     	     3	     3		                              
0x94	     	     3	     3		                              
0x97	     	     4	   188	*	DisplayImage                  
0x98	     	    10	    10		                              
0x99	     	     4	     4		                              
0x9a	     	     0	     0		                              
0x9b	     	     1	     1		                              
0x9d	     	     1	     1		                              
0x9e	     	     1	     1		                              
0x9f	     	     0	     0		EnableAnimationState		                              
0xa0	     	    16	    16		                              
0xaf	     	  1026	  1026		FirmwareUpdate                
0xb0	     	     8	    40	*	UnknownB0                     
0xb2	     	    16	    16		                              
0xb4	     	    21	    21		ObjectMoved                   
0xb5	     	     8	     8		ObjectStoppedMoving           
0xb6	     	    12	    12		ObjectTapped                  
0xb9	     	    10	    10		ObjectTapFiltered             
0xc2	     	     0	     0		RobotDelocalized              
0xc3	     	     0	     0		RobotPoked                    
0xc4	     	     1	     1		AcknowledgeAction            
0xc8	     	    29	    29		                              
0xc9	     	     6	     6		HardwareInfo                  
0xca	     	     1	     1		                              
0xcb	     	     1	     1		                              
0xcd	     	    12	  1036	*	NvStorageOpResult             
0xce	     	     9	     9		ObjectPowerLevel              
0xcf	     	     8	     8		                              
0xd0	     	    13	    13		ObjectConnectionState         
0xd1	     	     3	     3		                              
0xd2	     	    44	    44		                              
0xd7	     	     9	     9		ObjectUpAxisChanged           
0xec	     	     4	     4		                              
0xed	     	    12	    12		BodyInfo                      
0xee	     	   449	   449		FirmwareSignature             
0xef	     	     7	     7		FirmwareUpdateResult          
0xf0	    	    91	    91		RobotState                    
0xf1	    	    15	    15		AnimationState                
0xf2	    	    24	  1172	*	ImageChunk                    
0xf3	    	     9	     9		ObjectAvailable               
0xf4	    	    17	    17		ImageImuData

Connection Establishment

The engine sends a reset frame (0x01) to the robot with first_seq and seq set to 1 and ack set to 0.

The robot responds with a robot packets frame (0x09) with first_seq and seq set to 1 and ack set to 1, containing a connect packet (0x02). This establishes the connections.

The engine maintains the connection by periodically sending ping frames (0x0b). The robot responds with robot packet frames (0x09), containing a copy of the engine’s ping in a ping packet (0x0b). The pings have a sequence ID and a time stamp and allow the engine to measure round-trip time.

If the robot stops receiving ping frames for more than 5 s it will disconnect and display the message “COZMO 01”.

The engine can gracefully close the connection in one of two ways:

  • by sending a disconnect frame (0x03)
  • by sending an engine packets frame (0x07), containing a disconnect packet (0x03).

As long as a connection is established, the engine and the robot can exchange packets.

The engine sends packets in frames of types 0x04 and 0x07.

The robot sends packets in frames of type 0x09.

Capturing Cozmo Communication

Overview

Capturing the communication between the Cozmo app and Cozmo is very valuable for understanding how Cozmo works.

One way to achieve this is by placing a Linux machine between the two as shown on the following diagram.


+---------------+         Wi-Fi          +---------------+          Wi-Fi         +---------------+
|               |                        |               |                        |               |
| Mobile Device | ---------------------> | Linux Machine | ---------------------> |     Cozmo     |
|               |                  wlan1 |               | wlan0                  |               |
+---------------+                        +---------------+                        +---------------+

The Linux machine acts as a Wi-Fi client on one interface (wlan0) and associates with Cozmo. It acts as a Wi-Fi access point (AP) on the other interface and allows a mobile device, running the Cozmo app to associate with it.

With appropriate network configuration such a setup allows capturing Cozmo communication in pcap files using tcpdump.

Prerequisites

Connecting to Cozmo

Ensure that wireless tools and wpa_supplicant are installed.

$ sudo apt-get install wireless-tools wpasupplicant

Wake up Cozmo but placing it on the charging platform.

Make Cozmo display it’s Wi-Fi PSK key by rising and lowering its lift.

Get Cozmo’s Wi-Fi SSID by scanning for Wi-Fi devices:

$ sudo iwlist wlan0 scan
wlan0     Scan completed :
          Cell 01 - Address: 5E:CF:7F:XX:XX:XX
                    ESSID:"Cozmo_XXXXXX"
                    Protocol:IEEE 802.11bg
                    Mode:Master
                    Frequency:2.412 GHz (Channel 1)
                    Encryption key:on
                    Bit Rates:54 Mb/s
                    Extra:rsn_ie=30180100000fac020200000fac04000fac020100000fac020000
                    IE: IEEE 802.11i/WPA2 Version 1
                        Group Cipher : TKIP
                        Pairwise Ciphers (2) : CCMP TKIP
                        Authentication Suites (1) : PSK
                    Quality=100/100  Signal level=100/100

Open wpa_supplicant’s configuration file:

$ sudo vi /etc/wpa_supplicant/wpa_supplicant.conf

Configure wpa_supplicant to automatically connect to Cozmo by adding the following:

network={
    ssid="Cozmo_XXXXXX"
    psk="XXXXXXXXXXXX"
}

Load the new configuration (or reboot):

$ sudo wpa_cli -i wlan0 reconfigure
OK

At this point the Linux machine should be associated with Cozmo:

$ ip addr
...
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 80:1f:02:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 172.31.1.172/24 brd 172.31.1.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::1d4b:9d3b:c6f0:f5b1/64 scope link 
       valid_lft forever preferred_lft forever

Cozmo should respond to ping:

$ ping 172.31.1.1
PING 172.31.1.1 (172.31.1.1) 56(84) bytes of data.
64 bytes from 172.31.1.1: icmp_seq=1 ttl=128 time=1.94 ms
64 bytes from 172.31.1.1: icmp_seq=2 ttl=128 time=2.28 ms
...

Masquerading as a Cozmo

Install hostapd and dnsmasq:

$ sudo apt-get install hostapd dnsmasq

Edit dhcpdcd`s configuration file:

$ sudo vi /etc/dhcpcd.conf

Disable wpa_supplicant on wlan1 and configure a static IP address by adding the following:

interface wlan1
nohook wpa_supplicant
static ip_address=192.168.50.1/24

Edit dnsmasq’s configuration file:

$ sudo vi /etc/dnsmasq.conf

Configure DHCP on wlan1 by adding the following:

interface=wlan1
dhcp-range=192.168.50.50,192.168.50.100,255.255.255.0,24h

Restart dnsmasq

$ sudo systemctl start dnsmasq

Create a configuration file for hostapd:

$ sudo vi /etc/hostapd/hostapd.conf

Configure a Wi-Fi AP with WPA2 PSK on wlan1 by adding the following:

interface=wlan1
hw_mode=g
channel=1
wmm_enabled=0
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP
ssid=Cozmo_111111
wpa_passphrase=XXXXXXXXXXXX

The SSID should be different from Cozmo’s SSID and should follow the form Cozmo_XXXXXX, where XXXXXX are upper-case hexadecimal digits as this is what the Cozmo app looks for.

The passphrase should consist of exactly 12 upper-case hexadecimal digits as this is what the Cozmo app expects.

Edit /etc/default/hostapd:

$ sudo vi /etc/default/hostapd

Configure the newly created configuration file:

DAEMON_CONF="/etc/hostapd/hostapd.conf"

Enable and start hostapd:

$ sudo systemctl unmask hostapd
$ sudo systemctl enable hostapd
$ sudo systemctl start hostapd

Ensure that IP forwarding is enabled on boot:

$ sudo vi /etc/sysctl.conf

The following line should be uncommented:

net.ipv4.ip_forward=1

Ensure that IP forwarding is enabled:

$ sudo sysctl net.ipv4.ip_forward=1

The Cozmo app always tries to communicate with Cozmo using the IP address 172.31.1.1 .

Configure masquerading on wlan0 so that packets, coming from the Cozmo app, with source IP in the range 192.168.50.0/24, reach Cozmo with the wlan0 IP address of the Linux machine.

$ sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

This is necessary, because Cozmo only responds to UDP packets with source IP address in the range 172.31.1.0/24 .

Capturing Communication

Ensure that tcpdump is installed:

$ sudo apt-get install tcpdump

At this point, it should be possible to capture Cozmo communication using tcpdump:

$ sudo tcpdump -i wlan0 -w cozmo.pcap

Connect to cozmo from the app. The app should find at least 2 Cozmos (one being the masqueraded Linux machine) and a selection screen should show up.

The captured PCAP file can be analyzed with Wireshark or with pycozmo_dump.py.

Cozmo Functions

Overview

Cozmo is a complex distributed embedded system with the following main parts:

  • robot
  • cubes
  • charging platform

The robot can be subdivided into:

The Wi-Fi communication controller is responsible for the following functions:

  • Wi-Fi communication
  • over-the-air (OTA) firmware updates
  • NV RAM storage

Once Cozmo is powered on, the communications controller remains always powered on to maintain Wi-Fi communication.

On connection, the robot transmits its serial number with the HardwareInfo message and firmware version with the FirmwareSignature message.

The RTIP controller is responsible for:

  • OLED display image decoding
  • speaker audio decoding
  • camera image encoding
  • accelerometers
  • gyro

The body controller is in charge of:

  • left and right tread motors and encoders encoders
  • head motor and encoder
  • lift motor and encoder
  • backpack LEDs
  • backpack button (on newer models only)
  • Bluetooth LE communication (to cubes and charging platform)
  • IR LED
  • cliff sensor
  • batter charging

The body is powered on with the Enable message. The BodyInfo message communicates the body hardware version, serial number, and color.

Cubes use Nordic nRF31512 MCU. They are communicated with over Bluetooth LE and provide access to:

  • LEDs
  • Accelerometers
  • Battery voltage

Some charging platforms (aka “pads”) can be communicated with over Bluetooth LE. They contains 3 RGB LEDs that can be controlled, similar to cube LEDs.

The following sections provide more details on the use of each function.

Wi-Fi

Wi-Fi is activated automatically when the head board is powered on. The robot operates in access point (AP) mode.

cozmoclad defines a SetBodyRadioMode message that seems to allow changing the Wi-Fi channel but it is unclear how it can be used with the Cozmo protocol.

WifiOff Shutdown

Backpack LEDs

The 5 Backpack LEDs can be set controlled with 2 messages:

  • lightStateCenter - controls the top, middle, and bottom RGB LEDs.
  • LightStateSide - controls the left and right red-only LEDs.

Each color is defined by a 5-bit value for a total of 32768 colors.

See examples/backpack_lights.py for example usage.

Backpack Button

v1.5 and newer Cozmo models have a backpack button.

Button press and release events are communicated by the ButtonPressed message. It is immediately available on connection and does not require Enable to be used.

The RobotState message has a backpack_touch_sensor_raw field but it seems that it’s value does not change as a result of button presses.

See examples/events.py for example usage.

Wheels

The left and the right motor speeds can be controlled directly using the DriveWheels and TurnInPlaceAtSpeed messages. The motors can be stopped using the StopAllMotors message.

The actual speed of wheels is measured with Hall magnetic sensors. The values for each wheel can be read through the lwheel_speed_mmps and rwheel_speed_mmps fields of the RobotState message.

In addition, the and TurnInPlace message can be used to turn to a specific angle.

Localization

The robot maintains a world frame internally. It’s position and orientation with respect to it are transmitted every 30 ms or about 33 times per second with the RobotState message.

If the robot is unable to maintain correct position and orientation, for example because it is picked up or pushed, it will communicate this with a RobotDelocalized message.

The origin (0,0,0) of the world frame as well as “pose ID” can be set with the SetOrigin message. This is usually done on initial connection and and on receiving a RobotDelocalized message.

The timestamp in RobotState messages can be synchronized using the SyncTime message.

Path Tracking

The robot can traverse paths, composed of lines, arcs, and turns in place, described in world frame coordinates. The AppendPathSegLine, AppendPathSegArc, and AppendPathSegPointTurn messages can be used to build paths.

The last composed path can be executed using the ExecutePath message. One of it’s arguments can be used to request the reception of PathFollowingEvent message when path traversing finishes.

The status filed of the RobotState message has a robot_pathing flag that indicates whether the robot is currently traversing a path. The curr_path_segment filed indicates which segment is being traversed.

The ClearPath message can be used to destroy an already composed path. The TrimPath message can be used to delete path segments from the beginning or the end of a composed path.

See examples/path.py and examples/go_to_pose.py for example usage.

Lift

The head motor can be controlled directly, using the DriveLift and SetLiftHeight messages. SetLiftHeight is always followed by an AcknowledgeAction message before the lift start moving.

The actual lift height can be read through the lift_height_mm field of the RobotState message. The lift_inpos flag of the status field indicates whether the lift is in position or in motion.

The motor can be stopped using the StopAllMotors message.

The robot measures the angle of the lift, relative to its lowest possible position. It is calibrated similar to the head motor.

See examples/extremes.py for example usage.

OLED display

Images can be displayed on the robot’s OLED 128x64 display using the DisplayImage message. To reduce display burn-in, consecutive images are interleaved and only half of the display’s rows can be used at a time and the effective display resolution is 128x32.

The Cozmo protocol uses a special run-length encoding to compress images.

Display and audio are synchronized by audio messages (OutputAudio and OutputSilence).

AnimationState message which can be enabled using the EnableAnimationState message provide statistics on display usage.

See examples/display_image.py and examples/display_lines.py for example usage.

Speaker

The OutputAudio message can be used to transmit 744 audio samples at a time. The samples are 8-bit and u-law encoded.

Speaker volume can be adjusted with the SetRobotVolume message.

AnimationState message which can be enabled using the EnableAnimationState message provide statistics on audio usage.

See examples/audio.py for example usage.

Camera

Cozmo can send a stream of camera images in 320x240 (QVGA) resolution at a rate of ~15 frames per second.

The EnableCamera message enables camera image reception and the EnableColorImages message allows switching between grayscale and color images.

The camera gain, exposure time, and auto exposure can be controlled with the SetCameraParams message.

Images are encoded in JPEG format and transmitted as a series of ImageChunk messages. The header of the JPEG files is not transmitted to save bandwidth.

The ImageImuData message provides accelerometer readings at the time of capturing every image to allow for motion blur compensation.

See examples/camera.py for example usage.

IR LED

The IR LED (aka head light) can improve the camera performance in dark environments.

The IR LED can be turned on and off using the SetHeadLight message.

Accelerometers

The RobotState message communicates accelerometer readings which represent acceleration along the x, y, and z axes.

In addition, the robot automatically detects and communicates 2 types of events. The RobotPoked message is sent if the robot has been moved rapidly by an external force along the x or y axes. The FallingStarted and FallingStopped messages are send if the robot is moving rapidly along the z axis.

See examples/events.py for example usage.

Gyro

The RobotState message communicates gyro readings which represent angular velocity around the x, y, and z axes.

See examples/events.py for example usage.

Cliff Sensor

The robot has a “cliff sensor” that measures the distance to ground below the robot. This allows detecting cliffs and detecting when the robot is being picked up or put down.

The RobotState message communicates the raw cliff sensor readings.

In addition, the robot can be made to automatically stop when a cliff is detected with the EnableStopOnCliff message.

See examples/events.py for example usage.

Battery voltage

The RobotState message communicates raw battery voltage readings.

NV RAM Storage

The robot provides access to some amount of non-volatile memory (aka NV RAM) intended to store two main types of data:

  • unit-specific parameters (ex. camera calibration data and cube IDs)
  • mobile app data (ex. sparks and unlocked games and tricks)

The NV RAM storage is backed by the head’s ESP8266 controller external SPI flash. It is a NOR flash which drives the following specifics for its use:

  • an erase operation is needed before a write operation
  • data is erased in pages

The NvStorageOp message allows performing read, erase, and write operations. Data is addressed by the tag field and only the values enumerated by NvEntryTag can be used. Using any other address results in a NV_BAD_ARGS. Tags smaller than 0x80000000 are direct NOT flash memory addresses. Tags larger than 0x80000000 are virtual addresses that seem to be stored in the NVEntry_FactoryBaseTagWithBCOffset area.

NvStorageOpResult messages communicate results of NvStorageOp operations.

A backup through the mobile app, preserves the data behind the following keys:

  • NVEntry_GameSkillLevels
  • NVEntry_Onboarding
  • NVEntry_GameUnlocks
  • NVEntry_FaceEnrollData
  • NVEntry_FaceAlbumData
  • NVEntry_NurtureGameData
  • NVEntry_InventoryData
  • NVEntry_LabAssignments

See examples/nvram.py for example usage.

Firmware Updates

Cozmo firmware updates are distributed in “cozmo.safe” files that seem to contain firmware images for all three of Cozmos controllers - the Wi-Fi communication controller, the RTIP controller, and the body controller.

The “cozmo.safe” files start with a firmware signature in JSON format:

{
    "version": 2381,
    "git-rev": "408d28a7f6e68cbb5b29c1dcd8c8db2b38f9c8ce",
    "date": "Tue Jan  8 10:27:05 2019",
    "time": 1546972025,
    "messageEngineToRobotHash": "9e4a965ace4e09d86997b87ba14235d5",
    "messageRobotToEngineHash": "a259247f16231db440957215baba12ab",
    "build": "DEVELOPMENT",
    "wifiSig": "69ca03352e42143d340f0f7fac02ed8ff96ef10b",
    "rtipSig": "36574986d76144a70e9252ab633be4617a4bc661",
    "bodySig": "695b59eff43664acd1a5a956d08c682b3f8bd2c8"
}

This is the same signature, delivered with the FirmwareSignature message on initial connection establishment.

See docs/versions.md for more examples.

There seem to be individual signatures for each controller but the structure of the cozmo.safe files is not known.

The firmware image is transferred as-is from the engine to the robot, using FirmwareUpdate messages. It is divided into 1024 B chunks that are numbered consecutively, starting with 0. Each chunk is confirmed by the robot with a FirmwareUpdateResult message with status field set to 0.

Firmware transfer completion is indicated by the engine with e FirmwareUpdate message with chunk ID set to 0xFFFF and data set to all-zeros. The robot confirms firmware update completion by sending a FirmwareUpdateResult message that repeats the last chunk ID and has a status field set to 10.

Bluetooth LE

“Objects”, that can be connected to over Bluetooth LE announce their availability with an ObjectAvailable message periodically. The ObjectAvailable message contains the object type (e.g. light cube 1, 2, 3 or charging pad) and the object factory ID which identifies it uniquely.

The ObjectConnect message is used to initiate or terminate a connection to objects, using their factory ID.

Connection establishment and termination is announced with the ObjectConnectionState message. It contains a temporary “object ID” that is used to identify the object for the duration of the connection with it.

Cube LEDs

Cubes have 4 RGB LEDs that can be controlled individually.

A cube has to be “selected” first, using the CubeId message. A subsequent CubeLights message sets the state of all 4 cube LEDs.

Cubes can be programmed to perform simple LED light animations autonomously using the LightState structure and the CubeId.rotation_period_frames field.

See examples/cube_lights.py and examples/cube_light_animation.py for example usage.

Cube Battery Voltage

Cube battery voltage is communicated periodically with ObjectPowerLevel messages.

Cube Accelerometers

Cube accelerometer value reception can be enabled with the StreamObjectAccel message and are communicated every 30 ms with the ObjectAccel message.

In addition, the robot performs basic cube accelerometer ata processing and provides basic events with the following messages:

  • ObjectMoved
  • ObjectStoppedMoving
  • ObjectUpAxisChnaged
  • ObjectTapped
  • ObjectTapFiltered

Animations

To play animations, AnimationState message have to be enabled first using the EnableAnimationState message.

Animations are controlled with the StartAnimation, EndAnimation, and AbortAnimation messages.

Keyframes are transferred with the AnimHead, AnimLift, AnimBody, AnimBackpackLights, RecordHeading, TurnToRecordedHeading, and OutputAudio messages.

See examples/anim.py for example usage.

Cozmo Off-Board Functions

Cozmo mobile application resources consist of:

  • audio files
  • animations
  • animation group descriptions
  • behaviors
  • reaction triggers
  • emotions
  • activities
  • text-to-speech models

Robot firmware images are also distributed as part of the app resources.

Directory structure

cozmo_resources/
    assets/
        animationGroupMaps/
        animationGroups/
        animations/
        cubeAnimationGroupMaps/
        faceAnimations/
        RewardedActions/
    config/
        engine/
            animations/
            behaviorSystem/
                activities/
                behaviors/
            emotionevents/
            firmware/
            lights/
                backpackLights/
                cubeLights/
    sound/
        English(US)
    tts/

Audio files

WEM files

BNK files

Animations

Cozmo “animations” allow animating the following aspects of the robot:

  • body movement
  • lift movement
  • head movement
  • face images
  • backpack LED animations
  • audio

Cozmo animations are series of keyframes, stored in binary files in FlatBuffers format. Animation data structures are declared in FlatBuffers format in files/cozmo/cozmo_resources/config/cozmo_anim.fbs . The animation files are available in the following directory of the Android mobile application:

files/cozmo/cozmo_resources/assets/animations

Face images are generated procedurally. They are described by 43 parameters - 5 for the face and 19 for each eye. The face as a whole can be translated, scaled, and rotated. Each individual eye can be translated, scaled, and rotated. The 4 corners of each eye can be controlled and each eye has a lower and upper lid.

The following presentation from Anki provides some background information on Cozmo animations:

Cozmo: Animation pipeline for a physical robot

Animation groups

Animation groups are sets of animations with the same purpose.

Behaviors

Behaviors can be thought of as small applications that perform a specific function using the robot client API.

Reactions

Reactions map robot events to behaviors.

Emotions

Emotions are modeled as value functions that change in one of the following ways:

  • over time, driven by a decay function
  • as a result of reactions
  • as a result of behaviors

Activities

Activities are sets of behaviors with a rule how to choose

Cozmo Firmware Versions

Cozmo firmware images can be found under com.anki.cozmo/files/cozmo/cozmo_resources/config/engine/firmware in the Cozmo app.

Production Versions

{
    "version": 2381,
    "git-rev": "408d28a7f6e68cbb5b29c1dcd8c8db2b38f9c8ce",
    "date": "Tue Jan  8 10:27:05 2019",
    "time": 1546972025,
    "messageEngineToRobotHash": "9e4a965ace4e09d86997b87ba14235d5",
    "messageRobotToEngineHash": "a259247f16231db440957215baba12ab",
    "build": "DEVELOPMENT",
    "wifiSig": "69ca03352e42143d340f0f7fac02ed8ff96ef10b",
    "rtipSig": "36574986d76144a70e9252ab633be4617a4bc661",
    "bodySig": "695b59eff43664acd1a5a956d08c682b3f8bd2c8"
}

{
    "version": 2380,
    "git-rev": "6ef227df0d64427f95cb943e01d8ac3956646e4d",
    "date": "Thu Dec 20 17:33:45 2018",
    "time": 1545356025,
    "messageEngineToRobotHash": "3aed3b94dbf19e11b2775ff980874213",
    "messageRobotToEngineHash": "c5a95cb6f44c1b89a42784d0c637fda8",
    "build": "DEVELOPMENT",
    "wifiSig": "8694122d7de45ee085c488274d28b69b7b1f2f44",
    "rtipSig": "8acba259c7b440dc0a3467ae73f262a224f036db",
    "bodySig": "14d4420c42432211ae4cda4f78a41841b03a6b40"
}

{
    "version": 2315,
    "git-rev": "d96caf034da1c4a33d70d2c1e3bc5732ec68594a",
    "date": "Thu Nov  9 15:37:45 2017",
    "time": 1510270665,
    "messageEngineToRobotHash": "5d963ecd52d4ae18af796f14f02a3f60",
    "messageRobotToEngineHash": "d07d1f4dea884725adefd33de221a49f",
    "build": "DEVELOPMENT",
    "wifiSig": "2749d9fb97a138aa7b56429c3a587baf6dadfb6f",
    "rtipSig": "0605ff5cd1f37cf75573caac3678ecba12b9bebe",
    "bodySig": "76dc76aa624fac230603101206d3a4e2e50e76cb"
}

{
    "version": 2313,
    "git-rev": "7381fe56705992ffd03bef1bb1a7b2e7258e9bc2",
    "date": "Tue Nov  7 21:13:04 2017",
    "time": 1510117984,
    "messageEngineToRobotHash": "838bbe94628fd10783e40f6b6b9874df",
    "messageRobotToEngineHash": "6ae9b7733e469f4fef89479d63e214ba",
    "build": "DEVELOPMENT",
    "wifiSig": "5bfbabc73e0ec5e20a072b6ab87b60da8a51310a",
    "rtipSig": "349d2224cc00e56ee50a5b4ecb905a5ba64c791d",
    "bodySig": "5ac6821655294e88b5fb852427bd99120af16fb5"
}

{
    "version": 2214,
    "git-rev": "c363ccc897bc3748d234f80c21e4c8a33757d063",
    "date": "Wed Aug  9 11:01:32 2017",
    "time": 1502301692,
    "messageEngineToRobotHash": "861bbc71828456c0f073c4464bdcb21e",
    "messageRobotToEngineHash": "2dc8419f768f6f3fd4843cbb0a86f7f7",
    "build": "DEVELOPMENT",
    "wifiSig": "da7eb444c13475eb67b0c13336b24021b8cf540f",
    "rtipSig": "4cba42517073e77967ce8c7340376713001b4d0a",
    "bodySig": "74a1776d1c6a4213ccfbb0ad2c4099eafdf7ad0c"
}

{
    "version": 2158,
    "git-rev": "44c8d8a1d3a2b09b54da0ff4b6ceee75ec66e267",
    "date": "Thu Jun 15 10:00:23 2017",
    "time": 1497546023,
    "messageEngineToRobotHash": "71beec8d11144f3a3718d2cc5ea602f3",
    "messageRobotToEngineHash": "4018b2e764ec08f5fcacdb6358847cb0",
    "build": "DEVELOPMENT",
    "wifiSig": "e3f4a7e29b76321e3563f50e0b09c89378b5dc97",
    "rtipSig": "64efe94218e8eaac3576f2405bc5f01f020b0b7a",
    "bodySig": "d0c34ed006c71abe45ac735e4bb68bf1153b082b"
}

{
    "version": 1889,
    "git-rev": "e541e4247376d7945fd42a82a826b443effbeff2",
    "date": "Thu Mar 23 17:15:50 2017",
    "time": 1490314550,
    "messageEngineToRobotHash": "7098b4a266c0ccc2102a61fda53b8999",
    "messageRobotToEngineHash": "9b83f21da9120fdeebfeabe84af81c32",
    "build": "DEVELOPMENT",
    "wifiSig": "266d1d4f91c5ee069e628550a0331e8b0eb90f2b",
    "rtipSig": "bc90e2949be66851fb7ac035f5de9b52ff69fd14",
    "bodySig": "ccbb209db374f21ef233945f1515a70b8fe43114"
}

{
    "version": 1859,
    "git-rev": "11a52d6a4f2c5d89cef7085b836e8d0f2525808b",
    "date": "Mon Mar 20 23:29:56 2017",
    "time": 1490077796,
    "messageEngineToRobotHash": "54195812be0de998a4ebde795364d62b",
    "messageRobotToEngineHash": "90d8f3273055624b8444fbcbef555ee8",
    "build": "DEVELOPMENT",
    "wifiSig": "79dca08e85f21311e5551e38ecf0d3dab6ce006f",
    "rtipSig": "72519cd2bfb11bc799915dd8506a67b0ae5186da",
    "bodySig": "8746362ebc89e6235e3da103b9e9c0133cc3d1c1"
}

{
    "version": 1299,
    "git-rev": "6ced81297ac14067662acbed79cecac7f5eacd28",
    "date": "Mon Nov 21 15:25:58 2016",
    "time": 1479770758,
    "messageEngineToRobotHash": "61879d8808f0308cd8ae6340ddfe06e6",
    "messageRobotToEngineHash": "5914fda0b97c7aadaf0e4d97fc72610f",
    "wifiSig": "6cd4d9263e7a5b5da9eedc33e32c8baeb04a0ea6",
    "rtipSig": "24591dd715955eef0c1c7f0d89b4b41c122cbb26",
    "bodySig": "412ce6fc22f7407cb2e87eaacee3e9c4d7ca47ea"
}

Factory Versions

Cozmo factory firmwares identify with large version numbers.

Seen on Cozmo with HW v1.5:

{
    "build": "FACTORY",
    "version": 10501,
    "date": "Fri Apr 14 20:28:21 2017",
    "time": 1492201868
}

{
    "build": "FACTORY",
    "version": 10502,
    "date": "Mon Aug  7 09:21:24 2017",
    "time": 1502122884
}

Seen on development Cozmo with HW v1.7:

{
    "build": "FACTORY",
    "version": 10700,
    "date": "Thu Mar 28 14:18:13 2019",
    "time": 1553807893
}

Cozmo Hardware Versions

Hardware Version 4

  • fall 2016
  • does not have a button
  • come with platforms with LEDs?
2020-09-23 19:12:56.567 pycozmo.general      INFO     Firmware version 2381.
2020-09-23 19:12:56.568 pycozmo.robot        INFO     hardware.revision: Hardware 1.0
2020-09-23 19:12:56.598 pycozmo.general      INFO     Body S/N 0x088xxxxx, HW version 4, color 0.

Hardware Version 5

  • fall 2017
  • has an off button (EU Certification)
  • observed to have factory firmware v10501
  • teardown - https://www.microcontrollertips.com/teardown-anki-cozmo-vector/
2020-09-26 12:31:32.421 pycozmo.general      INFO     Firmware version 2381.
2020-09-26 12:31:32.422 pycozmo.robot        INFO     hardware.revision: Hardware 1.5
2020-09-26 12:31:32.453 pycozmo.general      INFO     Body S/N 0x088xxxxx, HW version 5, color 3.

Hardware Version 6

  • fall 2018
  • has an off button (Japan certification)

Hardware Version 7

  • fall 2019
  • has an on/off button
  • observed with development units
  • observed to have factory firmware v10700
  • observed to report undocumented color “5”
2020-09-24 20:04:35.823 pycozmo.general      INFO     Firmware version 10700.
2020-09-24 20:04:35.831 pycozmo.robot        INFO     hardware.revision: Hardware 1.7
2020-09-24 20:04:35.856 pycozmo.general      INFO     Body S/N 0x088xxxxx, HW version 7, color 5.

ESP8266

The ESP8266 is the main Cozmo controller, responsible for Wi-Fi communication.

SPI Flash Memory Map

The SPI flash size is 2 MB.

The below memory map has been reconstructed based on a SPI flash memory dump and NvEntryTag values.

Offset      Length      Type    Description
---------------------------------------------------------------------------------
0x00000000  0x00001000  Code    Bootloader.

0x00001000  0x00001000  Data    Unknown. The first 4 bytes are the head serial number.
0x00002000  0x00001000  Data    Unknown.

0x00003000  0x0007b800  Code    Application image 1.
0x0007e800  0x00001800  Data    Application image 1 signature. See versions.md .

0x00080000  0x0005e000  Code    Recovery image / factory firmware.

0x000de000  0x00000030  Data    Birth certificate.
0x000de030  0x00021fd0  Data    Factory data.

0x00100000  0x00003000  Data    Unknown.

0x00103000  0x0007b800  Code    Application image 2    
0x0017e800  0x00001800  Data    Application image 2 signature. See versions.md .

0x00180000  0x00018000  Data    Application data.
0x00198000  0x00028000  Data    Empty.

0x001c0000  0x0001e000  Data    Factory reserved 1.
0x001de000  0x0001e000  Data    Factory reserved 2. Empty?

0x001fc000  0x00001000  Data    Unknown.
0x001fd000  0x00001000  Data    Wi-Fi configuration 1.
0x001fe000  0x00001000  Data    Wi-Fi configuration 2.
0x001ff000  0x00001000  Data    Unknown.

pycozmo package

pycozmo.audiokinetic.exception AudioKinetic WWise exceptions.
pycozmo.audiokinetic.soundbank AudioKinetic WWise SoundBank representation and reading.
pycozmo.audiokinetic.soundbanksinfo AudioKinetic WWise SoundbanksInfo.xml representation and reading.
pycozmo.audiokinetic.wem AudioKinetic WWise WEM file representation and reading.
pycozmo.expressions.expressions Facial expression definitions.
pycozmo.activity Activity representation and reading.
pycozmo.anim Animation clip representation, reading, and preprocessing.
pycozmo.anim_controller Animation controller for audio, image, and animation playback.
pycozmo.anim_encoder Reading and writing of Cozmo animations in FlatBuffers (.bin) and JSON format.
pycozmo.audio Cozmo audio encoding.
pycozmo.behavior Behavior representation and reading.
pycozmo.brain Brain class - high level behavior and emotion engine.
pycozmo.camera Camera image decoding.
pycozmo.client Cozmo protocol client and high-level API.
pycozmo.conn Cozmo protocol low-level client and server connection.
pycozmo.emotions Emotion representation and reading.
pycozmo.event Event declaration and dispatching.
pycozmo.exception Exception declarations.
pycozmo.filter ID filtering for logging.
pycozmo.frame Cozmo protocol frame representation and encoding and decoding.
pycozmo.image_encoder Cozmo image run-length encoding and decoding.
pycozmo.lights Helper routines for working with colors and lights.
pycozmo.logging
pycozmo.object Cozmo objects (cubes, platforms, etc.).
pycozmo.procedural_face Cozmo procedural face rendering.
pycozmo.protocol_ast Cozmo protocol abstract syntax tree (AST) types.
pycozmo.protocol_base Cozmo protocol implementation base.
pycozmo.protocol_declaration Cozmo protocol abstract syntax tree (AST) declaration.
pycozmo.protocol_encoder Cozmo protocol packet encoder classes, based on protocol version 2381.
pycozmo.protocol_generator Cozmo protocol packet encoder code generator.
pycozmo.protocol_utils Cozmo protocol encoding helper classes and functions.
pycozmo.robot Robot constants and helper code.
pycozmo.robot_debug Cozmo firmware debug message decoding.
pycozmo.run Helper functions for running PyCozmo applications.
pycozmo.util Utility classes and functions.
pycozmo.window Cozmo protocol sliding window implementation.

Indices and tables