ESP32 Sketch data upload for Arduino IDE. Upload SPIFFS files to flash memory

4 September 2020
No Comments
SPIFFS, LittleFS
7 minutes

The ESP32 Sketch Data Upload plugin for the Arduino IDE allows you to upload files to the ESP32 memory area reserved for the file system (FS) managed using the SPIFFS file system. For the moment LittleFS is not yet officially supported by Espressif on the ESP32 platform.

The uploaded files must be managed using the FS.h library which supports the SPIFFS file system.

The size of the flash memory varies depending on the ESP32 module on board the development board. Recent modules generally have a 4MB flash memory of which 1MB, 2MB or 3MB can be allocated to the file system (File System – FS).

Using LittleFS on ESP32

The LittleFS system is not yet officially supported on ESP32. You your project necessarily requires the support of LittleFS, there are however several projects

Lorol has developed an ESP32 adaptation of the LittleFS Mbed library.

Warning, this project can only be used on Windows because it requires an external executable mklittlefs.exe

Project page on GitHub

The Illucius project is a cross-platform project.

Only downside, it already dates from 2017 and no longer seems to receive regular corrections.

Project page on GitHub

Joltwallet’s esp_littlefs project works with the ESP-IDF SDK

For those who develop directly with the API of the ESP-IDF SDK, you can test this project.

Project page on GitHub

As usual, you will need to install a plugin on the Arduino IDE to upload files in LittleFS format. It is here.

How to organize the files of an ESP32 project with SPIFFS?

Here is an example of the file tree of an ESP32 project whose HTML interface code is separate from the Arduino code. In general, the files of a WEB server are stored in a folder named www.

www/
├── index.html
├── style.css
├── code.js
code.ino

Apart from the Arduino code files (.ino, .cpp, .h files), all the other files must be moved to a data folder. The Arduino project tree becomes this

It is not possible to create subfolders in the data directory. However, during the transfer, the files will be “flattened”, ie the subfolders will not be created, instead the file will take the path as name. The “/” separator will be included in the file name. Be careful not to exceed 31 useful characters.

for example

data/
├── subfolder
   ├── index.hmtl

becomes

/data/subfolder/index.hmtl

You may quickly run into problems if you exceed 31 useful characters. Therefore, it is better to limit the depth of the tree … or not to create one.

Install ESP32 Sketch Data Upload tools for Arduino IDE

1 Before you begin, you need to install the ESP32 SDK from Espressif on the Arduino IDE. Everything is explained in detail in this article.

Choose the size allocated to the filesystem (partition scheme)

It is possible to allocate a certain amount of the flash memory of the ESP32 to the file system (FS) like on the ESP8266. By default, the framework allocates portions of memory according to a table called Partition Table (or Partition Scheme on the Arduino IDE).

Espressif has set some default schemas (on this GitHub page), but any board maker can change it.

This is the default Partition Table (source file). As you can see, the SPIFFS area is at the end so that it can take up the most of the available space.

Area	Type	Sub-type	Offset	Size
nvs	data	nvs	0x9000	0x5000
otadata	data	ota	0xe000	0x2000
app0	app	ota_0	0x10000	0x140000
app1	app	ota_1	0x150000	0x140000
spiffs	data	spiffs	0x290000	0x170000

The size that can be allocated depends on the size of the flash memory (4MB or 16MB) and on each manufacturer.

Select your development board from the list then open the Partition Scheme menu.

For a LoLin D32 Pro, 4 schemes are possible

Default advised
Minimal 2MB of Flash memory
No OTA
Minimal SPIFFS

I advise you to leave the default scheme especially if you want to set up a wireless update mechanism (OTA) in your project.

If this organization does not suit your development, you can switch to PlatformIO which allows you to finely define the Partition Table but using a csv file. More information here.

A test program with SPIFFS

On ESP32, the FS.h library has been renamed SPIFFS.h. The source code of the library is available on GitHub.

Create a new sketch, paste the code below and save

#include "SPIFFS.h"
 
void setup() {
  Serial.begin(115200);

  // Launch SPIFFS file system 
  if(!SPIFFS.begin()){
    Serial.println("An Error has occurred while mounting SPIFFS");
    return;
  }

  // Open test file in read only mode 
  File file = SPIFFS.open("/test.txt", "r");
  if(!file){
    // File not found | le fichier de test n'existe pas
    Serial.println("Failed to open test file");
    return;
  }

  // Display file content on serial port 
  Serial.println();
  Serial.println("Read test.txt file content:");
  while(file.available()){
    Serial.write(file.read());
  }
  file.close();
}
 
void loop() {
}

What does this code do?

We declare the SPIFFS.h library which allows access to the memory area using the SPIFFS file system.

#include "SPIFFS.h"

We start the SPIFFS file system. The error is reported in the event of a problem. Here, the program stops in the event of an error.

if(!SPIFFS.begin()){
  Serial.println("An Error has occurred while mounting SPIFFS");
  return;
}

We open the test.txt file with the open(filename, option) method in write only by specifying the option “r” (for read only). Use the “w” option to be able to write (and write) to a file.

File file = SPIFFS.open("/test.txt", "r");

We send the contents of the text file to the serial port

while(file.available()){ 
  Serial.write(file.read()); 
}

And we close the file

file.close();

Add files to the data folder

There is a shortcut allowing you to directly open the Arduino project folder from the Sketch menu -> Show sketch folder

Create a text file with any text editor and paste a “Hello World” or any text of your choice.

Save the file with the name test.txt.

Upload files to the ESP32 memory area

Warning, before uploading files to the memory area, the serial monitor and the graph plotter must be closed.

To upload the files saved in the data folder, simply launch the tool from the Tools -> ESP32 Sketch Data Upload menu

The operation only takes a few seconds

[SPIFFS] data   : /Users/diyprojects/Documents/Arduino/diyprojects/ESP32/SPIFFS/ESP32_SPIFFS_DEMO/data
[SPIFFS] start  : 2691072
[SPIFFS] size   : 1468
[SPIFFS] page   : 256
[SPIFFS] block  : 4096
/test.txt
[SPIFFS] upload : /var/folders/x_/w_k_y_ys531cxjfyvqpk1bwc0000gn/T/arduino_build_298507/ESP32_SPIFFS_DEMO.spiffs.bin
[SPIFFS] address: 2691072
[SPIFFS] port   : /dev/cu.usbserial-1420
[SPIFFS] speed  : 115200
[SPIFFS] mode   : dio
[SPIFFS] freq   : 80m

esptool.py v2.6
Serial port /dev/cu.usbserial-1420
Connecting.....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, BLK3 partially reserved, Coding Scheme 3/4
MAC: b4:e6:2e:88:xx:xx
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 1503232 bytes to 2848...

Writing at 0x00291000... (100 %)
Wrote 1503232 bytes (2848 compressed) at 0x00291000 in 0.3 seconds (effective 42700.6 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Note. Each time the files in the data folder are modified, they will have to be uploaded again manually.

Open the serial monitor and RESET the ESP32 module. This is what you should see

It is quite possible to access the SPIFFS file system on ESP32 through an FTP client. This is especially useful when developing projects with an HTML interface and you do not necessarily want to go through the Arduino IDE every time you want to upload the files.

Read, write, add data to a file by programming

In most cases, we will need to access and manipulate (write, add data, rename, delete …) files directly by programming with Arduino code. This is possible with the SPIFFS.h library presented in this article

Common issues

Here is a list of some common errors.

SPIFFS Upload failed!

If you get such a message, just close the serial monitor window.

esptool.py v2.6
Serial port /dev/cu.usbserial-1420
Traceback (most recent call last):
  File "esptool.py", line 2959, in 
  File "esptool.py", line 2952, in _main
  File "esptool.py", line 2652, in main
  File "esptool.py", line 222, in __init__
  File "serial/__init__.py", line 88, in serial_for_url
  File "serial/serialposix.py", line 268, in open
serial.serialutil.SerialException: [Errno 16] could not open port /dev/cu.usbserial-1420: [Errno 16] Resource busy: '/dev/cu.usbserial-1420'
Failed to execute script esptool
SPIFFS Upload failed!

A fatal error occurred: Timed out waiting for packet content

If you get such a message

esptool.py v2.6
Serial port /dev/cu.usbserial-1420
Connecting.....
Chip is ESP32D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, BLK3 partially reserved, Coding Scheme 3/4
MAC: b6:e6:2e:78:0b:ae
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...

A fatal error occurred: Timed out waiting for packet content

You just need to decrease the serial port speed to upload. Lower the speed to 115200 baud, then go back up to find out the maximum speed supported by the development board.