#Test Zigbee2MQTT project, hack of the Xiaomo Aqara Smart Home Gateway. Part 1: assembly, demo with Node-RED, 3D printed case [update]

The Zigbee2mqtt project that allows you to build a home automation bridge using a Texas Instrument Zigbee CC2531 sniffer. It is a project equivalent to the French project Zigate. It is compatible with most Xiaomi Smart Home Aqara and Mijia home automation accessories as well as some Ikea bulbs. After about 3 weeks of waiting, I received all the necessary equipment a few days ago (CC Debugger, Sniffer CC2531 and GBAN cabling interface).

 

I propose a real size test with some accessories Xiaomi Mijia. In addition to many benefits, the project Zigbee2MQTT will allow us to get rid of the official gateway and keep your hand on your personal data. All measurements, reports, orders will stay at home. The source code is fully available on GitHub.

Hardware needed to make the Xiaomi Zigbee Gateway

The zigbee2mqtt bridge uses a sniffer based on the Texas Instruments CC2531 (about $5) circuit. It will buy a Debugger CC from Texas Instrument (about $18) to install and update the firmware on the sniffer. It will also buy a specific cable (GBAN) that connects the CC Debugger Sniffer CC2531 (about $3). every time you want to update the firmware.

Hardware needed to operate the Zigbee2MQTT bridge

Once the firmware is installed on the CC2531 sniffer, you will no longer need the CC Debugger and the GBAN interface. It is an investment of about 20 € to make the gateway Zigbee but nothing prevents you to make several Gateway. By using a Raspberry Pi Zero or an Orange Pi Zero in Ethernet you can easily increase the Zigbee coverage in your home or why not your garden.

Here we will use a Raspberry Pi 3 equipped with a 16GB SD card on which Node-RED, Domoticz and the MQTT broker are installed. Node-RED will be used to integrate Zigbee devices into the Homekit application on iOS.

Principle of operation of the Zigbee2MQTT bridge

All actions, states, measures are thus intercepted and published on a MQTT broker (server). The gateway is bi-directional, ie we can also control the Zigbee devices by sending MQTT messages.

architecture zigbee mqtt xiaomi aqara

The project was developed in javascript using Nodejs. In theory, we will be able to operate the gateway on PC Windows / Linux, MAC and ARM mini PC such as the Raspberry Pi or Orange Pi For this tutorial, we will deploy it on a Raspberry Pi 3 on which is also installed a broker MQTT (Mosquitto), Node-RED and Domoticz.

Installing the firmware on the sniffer CC2531

The installation steps are few and very well explained on the official WiKi. I did not encounter any particular difficulty. The Wiki offers several installation methods for each system (Linux, macOS), but the easiest way is to use a Windows PC.

Install SmartRF Flash Programmer

Start by grabbing version 1 of the SmartRF Flash Programmer software from the Texas Instrument site by going to this page. You will need to create a free user account. Unzip the ZIP and install SmartRF Flash Programmer.

install SmartRF Flash Programmer cc debugger cc2531 zigbee mqttPlug the CC Debugger and the CC2531 Sniffer

Connect the Sniffer to the CC Debugger using a jumper cable. The Sniffer is equipped with a 1mm pitch connector. The GBAN card is just used to change the cable format. There is no key on the CC2531. Use the photo below to correctly orient the cable (the tamper inside the Sniffer).

cc2531 usb sniffer connected to coucou debugger

Connect the CC Debugger and CC2531 Sniffer to the computer. The LED of the Debugger will be red when connected., Press the RESET button to change it to green. We are ready to flash the firwmare.

cc debugger green led

Install drivers

Download the Windows driver swrc212 from the CC Debugger (here). Unzip the ZIP file. Plug in the CC Debugger. Windows will try to automatically install the driver but will probably fail. The easiest way is to open the device manager and right click on the CC Debugger and choose the option Update driver …

nstall drivers windows cc debugger sniffer cc2531 zigbee xiaomi

In the window that opens, choose Find a driver on my computer

install manually coucou debugger driver windows zigbee

Then, specify the path to the swrc212 folder that contains the drivers (32-bit and 64-bit).

path cebal swrc212 cc debugger zigbee xiaomi

Windows scan the directory looking for the correct drivers for your version. Validate the drivers found to complete the installation. At the end, both devices must be properly installed.

Download firmware

The Zigbee2MQTT firmware is available on GitHub on this page. Download the latest version (hex file)

Flash the firmware

Launch the SmartRF Flash Programmer program. Enter the path to the firmware image (previously downloaded hex file) and click Perform Action. The installation lasts less than a minute. Do not disconnect anything during the operation.

zigbee2mqtt flash firmware cc debugger cc2531 sniffer

Install the Zigbee2MQTT Gateway on the Raspberry Pi

Plug the USB sniffer into the Raspberry Pi and open a Terminal (or connect to SSH). Run this command to find the port on which the Sniffer is located.

ls -l /dev/tty*

 

In general, it is plugged into port ttyACM0.

Check now that Nodejs and npm are installed on Raspbian. If you use Jeedom or Node-RED, you have nothing to install.

node --version # v8.X.X mini
npm --version # v5.X.X mini

 

If this is not the case, run these two commands or follow this tutorial for more details.

sudo curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - 
sudo apt-get install -y nodejs git make g++ gcc

 

Now all you have to do is install Zigbee2MQTT script

# Clone zigbee2mqtt
sudo git clone https://github.com/Koenkk/zigbee2mqtt.git /opt/zigbee2mqtt
sudo chown -R pi:pi /opt/zigbee2mqtt

# Install dependencies
cd /opt/zigbee2mqtt
npm install

 

The npm install command remains pending at the end of the installation. You can leave it manually when reading the message

node-pre-gyp info ok

 

Configure the gateway

The configuration file is located in the data folder.

nano /opt/zigbee2mqtt/data/configuration.yaml
Modify the MQTT settings:
  • basic topic
  • IP address of the MQTT broker (server). Do not change anything if the sniffer is plugged into the same Raspberry Pi
  • user and password if the broker is protected by a password (highly recommended, follow this tutorial to configure it)
# MQTT settings
mqtt:
  # MQTT base topic for zigbee2mqtt MQTT messages
  base_topic: zigbee2mqtt
  # MQTT server URL
  server: 'mqtt://localhost'
  # MQTT server authentication, uncomment if required:
  # user: my_user
  # password: my_password

 

Save with CTRL + X then O. Everything is ready!

Add a Xiaomi Aqara or Mijia accessory

Start the gateway script

cd /opt/zigbee2mqtt
npm start

 

Wait for the Zigbee message: allowing new devices to join.

For the test, I propose to integrate the presence detector Xiaomi Mi Motion Sensor. The procedure is identical whatever the accessory.

Take a paperclip and hold the microswitch for 3 to 5 seconds until the blue LED flashes. You should get the unique identifier of your accessories directly. The script handles the pairing. Unlike the official gateway, the accessory must be prevented from falling asleep by pressing the association button regularly (once every second) until the association process begins.

As soon as the first message arrives (motion detection, temperature measurement …), it’s over, your Zigbee accessory is now associated.

2018-6-4 12:37:41 INFO Zigbee: allowing new devices to join.
2018-6-4 12:37:48 WARN Message without device!
spinlock: false []
2018-6-4 12:38:04 INFO New device with address 0x00158d000155d2b7 connected!
2018-6-4 12:38:04 INFO MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"device_connected","message":"0x00158d000155d2b7"}'
2018-6-4 12:38:08 INFO Connecting with device...
2018-6-4 12:38:08 INFO MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"connecting with device"}'
2018-6-4 12:38:13 INFO Connecting with device...
2018-6-4 12:38:40 INFO MQTT publish, topic: 'zigbee2mqtt/bridge/log', payload: '{"type":"pairing","message":"device incoming"}'
2018-6-4 12:38:45 INFO MQTT publish, topic: 'zigbee2mqtt/0x00158d000155d2b7', payload: '{"click":"double"}'

Rename accessories

By default, the script uses the unique identifier of the home automation accessory as its name. This is not necessarily the easiest way to navigate. You can easily change the name by going to change it in the configuration file. Stop the script and open the configuration file again

nano /opt/zigbee2mqtt/data/configuration.yaml

 

At the bottom of the configuration file, you will find a new section named devices. Indicate the name of each accessory with the key friendly_name. Avoid putting spaces in the name of the accessories to avoid Topic problems with MQTT.

devices:
  '0x00158d000155d2b7':
    friendly_name: 'Aqara_MiSwitch'
    retain: false
  '0x00158d000159a218':
    friendly_name: 'PIR'
    retain: false
  '0x00158d000149a352':
    friendly_name: 'Porte'
    retain: false

Automatically start the Zigbee2mqtt bridge when starting Raspbian with systemctl

Jens Willmer, the developer behind the project has released the script needed to start the bridge at startup Raspbian. In case of power failure or restart of the Raspberry Pi after an update, it is more convenient. Create a new systemctl configuration file

sudo nano /etc/systemd/system/zigbee2mqtt.service

Then past the following code

[Unit]
Description=zigbee2mqtt
After=network.target

[Service]
ExecStart=/usr/bin/npm start
WorkingDirectory=/opt/zigbee2mqtt
StandardOutput=inherit
StandardError=inherit
Restart=always
User=pi

[Install]
WantedBy=multi-user.target

 

Start manually the bridge using this command

sudo systemctl start zigbee2mqtt

 

Then check if all is working correctly using the status command

sudo systemctl status zigbee2mqtt

 

The service is correctly started as you can see

 zigbee2mqtt.service - zigbee2mqtt
   Loaded: loaded (/etc/systemd/system/zigbee2mqtt.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-09-11 11:45:24 UTC; 7s ago
   Main PID: 18086 (npm)
   CGroup: /system.slice/zigbee2mqtt.service
           ├─18086 npm
           ├─18095 /usr/bin/node /usr/lib/node_modules/npm/node_modules/update-notifier/check.js {"pkg":{"name":"npm","version":"5.6.0"}}
           ├─18109 sh -c node index.js
           └─18110 node index.js
sept. 11 11:45:24 raspberrypi systemd[1]: Started zigbee2mqtt.
sept. 11 11:45:29 raspberrypi npm[18086]: > zigbee2mqtt@0.1.0 start /opt/zigbee2mqtt
sept. 11 11:45:29 raspberrypi npm[18086]: > node index.js

 

Automatic start can be activated at the next start of Raspbian

sudo systemctl enable zigbee2mqtt.service

Other useful commands

  • sudo systemctl stop zigbee2mqtt, to stop the service. Update, rename the devices …
  • sudo journalctl -u zigbee2mqtt.service -f, display the execution log

Update of Zigbee2mqtt

The update is not what is simplest but it is enough to follow to the letter the orders indicated with Jens

# Stop zigbee2mqtt and go to directory
sudo systemctl stop zigbee2mqtt
cd /opt/zigbee2mqtt

# Backup configuration
cp -R data data-backup

# Update
git checkout HEAD -- npm-shrinkwrap.json
git pull
rm -rf node_modules
npm install

# Restore configuration
cp -R data-backup/* data
rm -rf data-backup

# Start zigbee2mqtt
sudo systemctl start zigbee2mqtt

An alarm system in 5 minutes with Node-RED and the Homekit plugin

While waiting for a complete test with Jeedom and Domoticz, here is a small demo realized in a few minutes with Node-RED. We will create a small alarm system using Node-RED to recover the states of a door contactor (for the entrance door) as well as a presence detector that monitors the living room.

A function node extracts the data recovered by the Zigbee bridge. Depending on what the sensor sends, we can recover the condition and sometimes the level and voltage of the battery. We recover at the output of the Node function 4 useful data:

  1. The state of the sensor (Boolean)
  2. The battery level (0 to 100%)
  3. A message that can be displayed on the dashboard or a stealth message
  4. The configuration for a Homekit accessory
msg1={};
msg2={};
msg3={};
msg4={};
msg1.payload = msg.payload.contact;
msg2.payload = msg.payload.battery;
if ( msg.payload.contact ) {
    msg3.payload = "La porte est fermée"
    msg4.payload = {
      ContactSensorState : 0
    }
} else {
    msg3.payload = "La porte est ouverte !"
    msg4.payload = {
      ContactSensorState : 1
    }
}

return [msg1,msg2,msg3,msg4];

And now, in less than 5 minutes without needing to install a home automation server, you can develop a small alarm system with Node-RED

Xiaomi smart home aqara mijia zigbee2mqtt node-red dashboard

The project flow to import into Node-RED

[{“id”:”1c34793f.a5ff77″,”type”:”mqtt in”,”z”:”82ba45a3.b704f8″,”name”:”PIR”,”topic”:”zigbee2mqtt/PIR”,”qos”:”2″,”broker”:”12f512eb.9c8b5d”,”x”:130,”y”:100,”wires”:[[“f3413bab.47fab8”]]},{“id”:”3a4b5967.b574c6″,”type”:”function”,”z”:”82ba45a3.b704f8″,”name”:”Split”,”func”:”msg1={};\nmsg2={};\nmsg3={};\nmsg4={};\nmsg1.payload = msg.payload.occupancy;\nmsg2.payload = msg.payload.battery;\n\nif ( msg.payload.occupancy ) {\n msg3.payload = \”Présence détectée !\”\n msg4.payload = {\n OccupancyDetected : 1\n }\n} else {\n msg3.payload = \”la pièce est vide\”\n msg4.payload = {\n OccupancyDetected : 0\n }\n}\n\n\nreturn [msg1,msg2,msg3,msg4];\n\n”,”outputs”:4,”noerr”:0,”x”:380,”y”:100,”wires”:[[“3b83fee7.b1a2b2”],[“c23193fa.630a9”],[“3a2d5f81.eee14”],[“bc648e31.f6e3f”]],”inputLabels”:[“json”],”outputLabels”:[“Etat”,”NIveau batterie”,”Message notification”,”Etat Homekit”]},{“id”:”f3413bab.47fab8″,”type”:”json”,”z”:”82ba45a3.b704f8″,”name”:””,”property”:”payload”,”action”:””,”pretty”:false,”x”:250,”y”:100,”wires”:[[“3a4b5967.b574c6”]]},{“id”:”c23193fa.630a9″,”type”:”ui_gauge”,”z”:”82ba45a3.b704f8″,”name”:””,”group”:”a76cdc5d.be91a”,”order”:0,”width”:”4″,”height”:”4″,”gtype”:”gage”,”title”:”PIR”,”label”:”%”,”format”:”{{value}}”,”min”:0,”max”:”100″,”colors”:[“#b51220″,”#e6e600″,”#17ca0d”],”seg1″:””,”seg2″:””,”x”:550,”y”:120,”wires”:[]},{“id”:”2e447a4a.601166″,”type”:”ui_gauge”,”z”:”82ba45a3.b704f8″,”name”:””,”group”:”a76cdc5d.be91a”,”order”:0,”width”:”4″,”height”:”4″,”gtype”:”gage”,”title”:”Porte”,”label”:”%”,”format”:”{{value}}”,”min”:0,”max”:”100″,”colors”:[“#b51118″,”#e6e600″,”#32ca2b”],”seg1″:””,”seg2″:””,”x”:530,”y”:360,”wires”:[]},{“id”:”7cc7f334.508d0c”,”type”:”mqtt in”,”z”:”82ba45a3.b704f8″,”name”:”Porte”,”topic”:”zigbee2mqtt/Porte”,”qos”:”2″,”broker”:”12f512eb.9c8b5d”,”x”:130,”y”:340,”wires”:[[“1c94d73d.c53d69”]]},{“id”:”b832b436.0ac2a8″,”type”:”function”,”z”:”82ba45a3.b704f8″,”name”:”Split”,”func”:”msg1={};\nmsg2={};\nmsg3={};\nmsg4={};\nmsg1.payload = msg.payload.contact;\nif ( msg.payload.contact ) {\n msg3.payload = \”La porte est fermée\”\n msg4.payload = {\n ContactSensorState : 0\n }\n} else {\n msg3.payload = \”La porte est ouverte !\”\n msg4.payload = {\n ContactSensorState : 1\n }\n}\n\nmsg2.payload = msg.payload.battery;\n\n\n\nreturn [msg1,msg2,msg3,msg4];”,”outputs”:4,”noerr”:0,”x”:380,”y”:340,”wires”:[[“a1d7b7f8.8f6458”],[“2e447a4a.601166”],[“f37b5a94.a19b98″,”a47a5db8.951a4”],[“5d52112d.4c9f”]],”outputLabels”:[“Etat”,”Niveau batterie”,”Message Notification”,”Etat Homekit”]},{“id”:”1c94d73d.c53d69″,”type”:”json”,”z”:”82ba45a3.b704f8″,”name”:””,”property”:”payload”,”action”:””,”pretty”:false,”x”:250,”y”:340,”wires”:[[“b832b436.0ac2a8”]]},{“id”:”a1d7b7f8.8f6458″,”type”:”ui_switch”,”z”:”82ba45a3.b704f8″,”name”:””,”label”:”Etat Porte”,”group”:”80560196.e17ca”,”order”:1,”width”:0,”height”:0,”passthru”:true,”decouple”:”false”,”topic”:””,”style”:””,”onvalue”:”true”,”onvalueType”:”bool”,”onicon”:””,”oncolor”:””,”offvalue”:”false”,”offvalueType”:”bool”,”officon”:””,”offcolor”:””,”x”:540,”y”:320,”wires”:[[]]},{“id”:”3b83fee7.b1a2b2″,”type”:”ui_switch”,”z”:”82ba45a3.b704f8″,”name”:””,”label”:”Etat PIR”,”group”:”80560196.e17ca”,”order”:4,”width”:0,”height”:0,”passthru”:true,”decouple”:”false”,”topic”:””,”style”:””,”onvalue”:”true”,”onvalueType”:”bool”,”onicon”:””,”oncolor”:””,”offvalue”:”false”,”offvalueType”:”bool”,”officon”:””,”offcolor”:””,”x”:560,”y”:80,”wires”:[[]]},{“id”:”f37b5a94.a19b98″,”type”:”ui_toast”,”z”:”82ba45a3.b704f8″,”position”:”top right”,”displayTime”:”10″,”highlight”:””,”outputs”:0,”ok”:”OK”,”cancel”:””,”topic”:””,”name”:””,”x”:570,”y”:400,”wires”:[]},{“id”:”bc648e31.f6e3f”,”type”:”homekit-service”,”z”:”82ba45a3.b704f8″,”accessory”:”be545a0c.987228″,”name”:”PIR”,”serviceName”:”OccupancySensor”,”x”:550,”y”:200,”wires”:[[]]},{“id”:”5d52112d.4c9f”,”type”:”homekit-service”,”z”:”82ba45a3.b704f8″,”accessory”:”abf76c70.fbd85″,”name”:”Mi Switch”,”serviceName”:”ContactSensor”,”x”:540,”y”:480,”wires”:[[]]},{“id”:”a47a5db8.951a4″,”type”:”ui_text”,”z”:”82ba45a3.b704f8″,”group”:”80560196.e17ca”,”order”:2,”width”:0,”height”:0,”name”:””,”label”:”Etat”,”format”:”{{msg.payload}}”,”layout”:”row-spread”,”x”:530,”y”:440,”wires”:[]},{“id”:”3a2d5f81.eee14″,”type”:”ui_text”,”z”:”82ba45a3.b704f8″,”group”:”80560196.e17ca”,”order”:3,”width”:0,”height”:0,”name”:””,”label”:”Présence”,”format”:”{{msg.payload}}”,”layout”:”row-spread”,”x”:560,”y”:160,”wires”:[]},{“id”:”12f512eb.9c8b5d”,”type”:”mqtt-broker”,”z”:””,”name”:”MQTT Local”,”broker”:”localhost”,”port”:”1883″,”clientid”:””,”usetls”:false,”compatmode”:true,”keepalive”:”60″,”cleansession”:true,”birthTopic”:””,”birthQos”:”0″,”birthPayload”:””,”closeTopic”:””,”closeQos”:”0″,”closePayload”:””,”willTopic”:””,”willQos”:”0″,”willPayload”:””},{“id”:”a76cdc5d.be91a”,”type”:”ui_group”,”z”:””,”name”:”Niveaux batteries”,”tab”:”6355179e.0ee2e8″,”order”:2,”disp”:true,”width”:”4″,”collapse”:false},{“id”:”80560196.e17ca”,”type”:”ui_group”,”z”:””,”name”:”Etats “,”tab”:”6355179e.0ee2e8″,”order”:1,”disp”:true,”width”:”8″,”collapse”:false},{“id”:”be545a0c.987228″,”type”:”homekit-accessory”,”z”:””,”accessoryName”:”Xiaomi Mi Motion”,”pinCode”:”111-11-111″,”port”:””,”manufacturer”:”Xiaomi”,”model”:”RTCGQ01LM”,”serialNo”:”0x00158d000159a218″,”accessoryType”:”11″},{“id”:”abf76c70.fbd85″,”type”:”homekit-accessory”,”z”:””,”accessoryName”:”Xiaomi Mi Switch”,”pinCode”:”111-11-111″,”port”:””,”manufacturer”:”Xiaomi”,”model”:”WXKG01LM”,”serialNo”:”0x00158d000155d2b7″,”accessoryType”:”6″},{“id”:”6355179e.0ee2e8″,”type”:”ui_tab”,”z”:””,”name”:”Xiaomi Smart Home”,”icon”:”dashboard”}]

 

You can even integrate your Xiaomi Aqara and Mijia accessories into the House app if you own an iPhone.

xiaomi smart home pir motion door aqara mijia zigbee2mqtt nodered

Security accessories can send notifications as soon as an event is triggered even if you’re not home (provided you have a Homekit bridge at home, an Apple TV or an iPad).

xiaomi smart home pir motion door aqara mijia zigbee2mqtt nodered homekit notification

To learn more, I advise you to read these previous tutorials:

Lock the gateway

Once all the accessories are associated, it is safer to forbid the association of new peripherals. In the configuration file, pass the permit_join key to false.

Case to print in 3D

I made available the STL files of a small box to print in 3D on Thingiverse here.

cc2531-zigbee2mqtt-gateway-xiaomi-aqara-mijia

Comparison of the reception range compared to the official gateway

The CC2531 USB sniffer is primarily an accessory for debugging connected object projects. The Zigbee antenna is engraved on the PCB. It is not necessarily optimized for very long range reception. To be clear, I made a comparison test (fast) between my official gateway and the Sniffer CC2531. Nothing really scientific in this test, I simply placed the two gateways next to each other and checked the reception of the connected button under the same conditions (in the same place).

Location Official Xiaomi Gateway CC2531 Zigbee Sniffer
Room 1 Oui Yes
Room 2 Non Yes
Room 3 Non No
Attic Oui Yes
Attic (very far) Non No

To my surprise, the Sniffer receives as well (if not better) the controls of the Mi Switch connected button than the official gateway. It’s a shame that there is no connector for an external antenna to further improve the reception range. But considering the price of a Sniffer, do not deprive yourself to multiply the bridges 🙂

Project GitHub Page: zigbee2mqtt

 

Subscribe to the weekly newsletter

No spam and no other use will be made of your email. You can unsubscribe anytime.

DIY Projects