How to store data on a micro SD card. Arduino code compatible ESP32, ESP8266

arduino esp32 esp8266 sd card reader
Share on facebook
Share on twitter
Share on linkedin
Share on pinterest
Share on email
Share on whatsapp

For connected object projects that require the storage of a large amount of data, it may be necessary to use micro SD card storage. Indeed, most Arduino development boards do not have Flash memory unlike ESP32 or ESP8266.

 

The ESP32 and ESP8266 development boards have a flash memory module of at least 4MB of which at least 3MB can be used to store data. Be careful because flash memory can be written 10,000 times which is low for a data acquisition system.

On the other hand, the on-board flash memory storage is very well suited for saving the source code of the HTML interface, style sheet, parameter files … To find out more, here are some articles on the subject

SPI bus pin identification of the most common development boards

All micro-SD card reader modules use the SPI bus to communicate with the microcontroller. Unable to list all development boards in the market.

However, here is a summary table of the most common cards. Some development boards have multiple SPI buses.

Bus SPIMOSIMISOSCKCS
Arduino Uno or Nano111121310
sd card reader arduino uno nano
Arduino Mega151505253
Arduino MKR Zero

Arduino MKR WiFi 1010

1D8D10D9
Arduino nano 331D11D12D13
ESP82661D7D6D5D8
esp8266 sd card reader
ESP3212319185
esp32 circuit sd card reader
STM32F1031PB5PB4PB3PA15
2PB15PB14PB13PB12

Breakout (expansion board) micro SD card reader on SPI bus

Micro SD card reader modules use the SPI bus to communicate with the micro-controller. The SPI bus will require 4 pins to be reserved for your project. This is not a problem for most development boards with the exception of the more limited ESP8266.

See more offers

In this case, you can opt for an ESP32 whose programming is very similar. You can also add an I2C expansion board that will increase the number of IO or analog inputs. Here are some common solutions.

Development boards with integrated SD card reader

Some development boards include a card reader. micro-SD. Here are some development boards that are very common in the general public.

Unfortunately no Arduino board (unless I’m mistaken) is equipped with an SD card reader, including the new Pro models. Ditto for the ESP8266. You will have to go through a stackable module or shield (in mini d1 format ).

Connect the micro-SD card directly to the micro-controller!

Each micro SD card has an SPI controller on board, so we can easily do without a micro-SD card reader!

It is much less elegant but you can help yourself with some resistors in case of failure or while waiting to receive your equipment. To learn more, read this tutorial offered by Renzo Mischianti.

SD Card Adapter arduino esp8266 esp32 direct connection

Source: mischianti.org

Format SD card as FAT16 or FAT32

The SD library only supports FAT16 or FAT32 type formatting. FAT16 and FAT32 file systems are supported by all operating systems. Windows, macOS, and Linux (including ARM versions). You will have no problem reading your data.

To avoid any compatibility problem, it is better to use the free SD Card Formatter utility developed by the SD association. SD Card Formatter is available for all platforms here .

No adjustment to make, everything is automatic!

sd card formatter esp8266 esp32 arduino

Functions offered by the SD.h library, SD and File classes

The official documentation is available online here .

SD class, initialization, folder operations

The SD class provides functions for accessing the SD card and manipulating the files and directories it contains.

begin(CS_pin)

Initializes the SD library and the card. Returns true on success, false on failure.

If the pin is not specified, the library uses the SS pin by default of the target platform

exists(filename)

Tests whether a file or directory exists on the SD card. Returns true if the file or directory exists, false otherwise.

end()

Closes communication with the card reader before ejection

mkdir(folder_name)

Create a directory on the SD card, including the tree structure if it does not exist. Returns true if the directory creation was successful, false otherwise.

For example, the command mkdir(“data/today”) will create the data folder then the today sub-folder .

open(file_path, mode)

Opens a file on the SD card.

The methods for performing file operations are listed in the next paragraph.

Mode (optional parameter), the mode in which to open the file

  • FILE_READ (default mode) opens the file for reading. The pointer is placed at the beginning of the file
  • FILE_WRITE opens the file for reading and writing. The pointer is placed at the end of the file. If the file does not exist, it is created automatically.

Warning, to be able to create a file automatically, the destination folder must already exist. Test the existence of the folder using the exists() function.

remove(filename)

Deletes a file from the SD card. Returns true if the file was deleted successfully, false if it failed. No return value if the file did not exist on the SD card.

rmdir(folder_name)

Deletes a directory from the SD card. Returns true if the directory removal was successful, false otherwise. No return value if the directory did not exist on the SD card.

Warning, the directory must be empty to be able to be deleted

File class, file operations

The File class is used to perform all read and write operations of individual files on the SD card.

name()
Returns the name of the file

available()

Checks if the file is not empty. Returns the number of bytes used by the file.

close()

Close the previously opened file with the open() function

flush()

Ensures that all bytes written to the file are physically saved to the SD card. This is done automatically when the file is closed.

peek()

Reads one byte of the file without going to the next. The pointer is not moved. Used the seek() method to move the pointer within the file.

position()

Retrieves the current position of the pointer in the file. Unsigned byte.

fprint(data, base)

Prints (saves) the data to the file, which must have been previously opened using the open() function in FILE_WRITE mode .

data   the data to print. Supported types: char, byte, int, long or string.

base (optional) the base in which to print the numbers: BIN for binary (base 2), DEC for decimal (base 10), OCT for octal (base 8), HEX for hexadecimal (base 16).

println(data, base)

Same as the print () method but adds a carriage return (CR) and new line (LF). Handy for a data logger.

seek(pos)

Set the pointer file to the pos ition indicated (unsigned long). The position must be between 0 and the file size (inclusive). Returns true on success, false on failure

size()

Get the file size in bytes (unsigned).

read() ou read(buf, len)

Read from file from file at pointer position.

Use the open() function before you can read from a file

If in doubt, use the isDirectory() method to find out if it is a file.

Use the seek() function to place the pointer at the desired location. Return the next byte (or character), or -1 if none are available.

Use the available() method to test if the file is not empty

Remember to close the file with close() as soon as there is nothing more to read.

write(data) ou write(buf, len)

Writes data to the file. Returns the number of bytes written.

Use the open() function before you can read from a file

Use the seek() function to place the pointer at the desired location. Return the next byte (or character), or -1 if none are available.

Remember to close the file with close() as soon as there is nothing more to read.

isDirectory()

Returns true if it is a directory.

openNextFile()

Return the next file or folder in the path.

rewindDirectory()

Returns to the first file in the directory, use in conjunction with openNextFile().

Availability of functions on Arduino, ESP32 and ESP8266

The SD.h and File.h libraries are available on the 3 platforms. One might think that the functionalities are identical whatever the platform, but this is not the case, which makes the developments a bit painful. It is the choice of Espressif who preferred to use the same names for these bookstores.

Here is a summary table to help you find your way around. Espressif went back for the ESP32 by offering the same functions as on Arduino. We just have 3 additional functions to determine the type, the space occupied and the space available on the micro SD card.

Avoid using specific functions to make the code cross-platform. ReadBytes(), ReadString(), cardType() … If this is not possible, use the #define directive to adapt the code for each target.

ArduinoESP8266ESP32
Source code SD.hGitHubGitHubGitHub
Source code File.hGitHubGitHubGitHub
begin()
end()
cardType()

Types

CARD_NONE

CARD_MMC

CARD_SD

CARD_SDHC

CARD_UNKNOWN

cardSize()
totalBytes()
usedBytes()
open()

FILE_READ

FILE_WRITE

FILE_READ

FILE_WRITE

FILE_READ

FILE_WRITE

FILE_APPEND

exists()File.h
mkdir()File.h
remove()File.h
rmdir()File.h
type() Librairie sd2cardsd2card library missing
fatType()sd2card librarysd2card library missing
File class
available()
close()
flush()
getCreationTime()  
isDirectory()
isFile()  
name()
open()
openNextFile()
rewindDirectory()
peek()
position()
read()
readBytes()  
readString()  
seek()
setTimeCallback()  
size()
truncate()  
write()
getLastWrite() 

Mount the SD card. Arduino code compatible ESP32, ESP8266

The first thing to do before you can read or save files to the SD card is to create an SD object using the begin () method .

Create a new sketch or PlatformIO project then upload the code below. You can use the platformio.ini configuration file to test on Arduino Uno, ESP32, or ESP8266.

Note, on PlatformIO, it seems necessary to manually include the SD.h (161) library. This is unnecessary on ESP8266 and ESP32.

Cross-platform Arduino codePlatformio.ini file
#include <Arduino.h>
#include <Wire.h>
#include <FS.h>
#include <SPI.h>
#include <SD.h>

#if defined(__AVR__)
  #define SD_CS    10
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
  #define SD_CS     53
#elif defined(ESP8266)
  #define SD_CS    D8
#elif ESP32
  #define SD_CS    5   //SS
#endif     

void setup(){
  Serial.begin(115200);
  
  if(!SD.begin(SD_CS)){
      Serial.println("Card Mount Failed");
      return;
  } else {
    Serial.println("SD Card mounted with success");
  }
}

void loop(){
}
[env:lolin_d32]
platform = espressif32
board = lolin_d32
framework = arduino
monitor_speed = 115200

[env:d1_mini_lite]
platform = espressif8266
board = d1_mini_lite
framework = arduino
monitor_speed = 115200

[env:uno]
platform = atmelavr
board = uno
framework = arduino
monitor_speed = 115200
lib_deps = 161

 

Explanation of the code

The SPI library is required to communicate with the SD card reader. The SD library provides SD and File classes for reading, writing and handling files on the SD card.

#include <SPI.h>
#include <SD.h>

We use the #define directive to specify to the compiler the code to include according to the target platform, here the CS pin necessary for the begin() function.

#if defined(__AVR__) 
  #define SD_CS 10 
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__) 
  #define SD_CS 53 
#elif defined(ESP8266)
  #define SD_CS D8 
#elif ESP32 
  #define SD_CS 5 //SS 
#endif

The SD.begin(SD_CS) method returns a boolean which indicates whether the SD object could be initialized, ie if a valid SD card is in the card reader.

ESP32 only, SD card type, occupied space and available space

Here is a small program which allows you to know the occupied space, the available space as well as the type of SD card inserted in the reader.

#include <Arduino.h>
#include <SPI.h>
#include <SD.h>

#if defined(__AVR__)
  #define SD_CS    10
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
  #define SD_CS     53
#elif defined(ESP8266)
  #define SD_CS    D8
#elif ESP32
  #define SD_CS    5   //SS
#endif     

void setup(){
  Serial.begin(115200);
  
  if(!SD.begin(SD_CS)){
      Serial.println("Card Mount Failed");
      return;
  } else {
    Serial.println("SD Card mounted with success");
    #if ESP32
      uint8_t cardType = SD.cardType();

      if(cardType == CARD_NONE){
          Serial.println("No SD card attached");
          return;
      }

      Serial.print("SD Card Type: ");
      if(cardType == CARD_MMC){
          Serial.println("MMC");
      } else if(cardType == CARD_SD){
          Serial.println("SDSC");
      } else if(cardType == CARD_SDHC){
          Serial.println("SDHC");
      } else {
          Serial.println("UNKNOWN");
      }

      uint64_t cardSize = SD.cardSize();
    int cardSizeInMB = cardSize/(1024 * 1024);
     
    Serial.printf("Card size: %u MB \n", cardSizeInMB);
 
    uint64_t bytesAvailable = SD.totalBytes(); 
    int spaceAvailableInMB = bytesAvailable/(1024 * 1024);
 
    Serial.printf("Space available: %u MB \n", spaceAvailableInMB);
 
    uint64_t spaceUsed = SD.usedBytes(); 
    Serial.printf("Space used: %u bytes", spaceUsed);

    #endif   
  }
}

void loop(){
}

Write to file, ESP32 version

Here is finally a last example that explains how to write data to a file. We will recover the temperature and atmospheric pressure on a BMP180 connected to the I2C bus.

esp32 sd card reader data logger bp180

Before uploading the project, you will need to modify the variables to suit your ESP32 board:

  • SD_CS pin CS
  • PIN_SDA SDA pin of the I2C bus, by default 21
  • PIN_SCL SCL pin of the I2C bus, by default 22

esp32 data logger bmp180 bmp280 sd card

By default, a measurement is made every minute (TIME_TO_SLEEP) before putting the ESP32 into deep standby.

#include <Arduino.h>
#include <Wire.h>
#include <FS.h>
#include <SPI.h>
#include <SD.h>
#include <Adafruit_BMP085.h>

#define SD_CS    5   
#define PIN_SDA 22
#define PIN_SCL 17 // PIN 21 is not exposed on LoLin D32 GPIO  

// Conversion factor for micro seconds to seconds
uint64_t uS_TO_S_FACTOR = 1000000;  
// Sleep for 1 minutes = 60 seconds
uint64_t TIME_TO_SLEEP = 60;

Adafruit_BMP085 bmp;
int BME_EXIST = false;   
float temp = 0;
float pressure = 0;

void recordNewData();
void storeDataToSDCard(fs::FS &fs, const char * path, const char * message);

void setup(){
  Serial.begin(115200);

  Wire.begin(PIN_SDA, PIN_SCL);

  if (!bmp.begin()) {
    Serial.println("Could not find BMP180 or BMP280 sensor");
  } else {
    BME_EXIST = true;
  }
  if(!SD.begin(SD_CS)){
    Serial.println("Card Mount Failed");
    return;
  } else {
    Serial.println("SD Card mounted with success");
  }
}

void loop(){
  if ( BME_EXIST ) {
    recordNewData();
  }
  
  // Enable Timer wake_up
  esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
  esp_deep_sleep_start();
}

void recordNewData(){
  temp = bmp.readTemperature();
  pressure = bmp.readSealevelPressure();
  String message = String(temp) + "," + String(pressure);
  storeDataToSDCard(SD, "/data.txt", message.c_str());
}

void storeDataToSDCard(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Appending data to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if(!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if(file.println(message)) {
    Serial.println("Data appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

Explanation of the code

Depending on the development board, the I2C pins may not be exposed on the GPIO. The Wire.h library for ESP32 and ESP8266 allows you to assign other pins by executing the function.

Wire.begin(PIN_SDA, PIN_SCL);

To learn more about the Wire.h library, read this article

The Adafruit_BMP085 library has the annoying tendency to crash the ESP32 if the BMP180 has not been properly initialized. It is easy to work around the problem by setting a flag to True if the BMP180 has been properly initialized.

if (!bmp.begin()) {
    Serial.println("Could not find BMP180 or BMP280 sensor");
} else {
    BME_EXIST = true;
}

We have the ESP32 dormant between two recordings

esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);
esp_deep_sleep_start();

To learn more about putting the ESP32 to sleep, read this detailed tutorial

We prepare the recording in the form of a String. Care will be taken to convert the digital data into a string.

We will use the println() method to add a record to the data file. To wrap between each record with the print() function , you must add the string “\ r \ n” .

The print() and print() methods only accept C ++ format strings. We must therefore convert the strings in this format using the c_str() method .

String message = String(temp) + "," + String(pressure);
storeDataToSDCard(SD, "/data.txt", message.c_str());

All functions on strings are explained in this tutorial

To save data to an existing file, the open() method for ESP32 has the FILE_APPEND option . On Arduino or ESP8266, we will simply open with the FILE_WRITE option .

If the file is correctly opened, the recording is added. Since the file pointer is automatically placed at the end of the file, the data is appended. Here we use the println() method which automatically jumps to the line.

To avoid damaging the SD card and freeing it for another function, we close the file with the close() method .

  File file = fs.open(path, FILE_APPEND);
  if(!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if(file.println(message)) {
    Serial.println("Data appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();

Example of the data file created

21.60,98569.00
21.60,98578.00
21.60,98570.00
21.60,98576.00
21.60,98570.00
21.60,98572.00
21.60,98572.00
21.60,98571.00
21.60,98572.00
21.60,98567.00
21.60,98569.00
21.60,98568.00
21.60,98571.00

Write to file, ESP8266 or Arduino version

The code adapted to the Arduino or the ESP8266

#include <Arduino.h>
#include <Wire.h>
#include <FS.h>
#include <SPI.h>
#include <SD.h>
#include <Adafruit_BMP085.h>  

#define TIME_TO_WAIT 5000
#define PATH_DATA_FILE "/data.txt"

Adafruit_BMP085 bmp;
int BME_EXIST = false;   
float temp = 0;
float pressure = 0;

#if defined(__AVR__)
  #define SD_CS    10
#elif defined(__AVR_ATmega1280__) || defined(__AVR_ATmega2560__)
  #define SD_CS    53
#elif defined(ESP8266)
  #define SD_CS    D8
#endif     

void setup(){
  Serial.begin(115200);

  if (!bmp.begin()) {
    Serial.println("Could not find BMP180 or BMP085 sensor at 0x77");
  } else {
    BME_EXIST = true;
  }

  if(!SD.begin(SD_CS)){
      Serial.println("Card Mount Failed");
      return;
  } else {
    Serial.println("SD Card mounted with success");
  } 
}

void loop(){
  if ( BME_EXIST ) {
    temp = bmp.readTemperature();
    pressure = bmp.readSealevelPressure();
    String message = String(temp) + "," + String(pressure) ;

    File file = SD.open(PATH_DATA_FILE, FILE_WRITE);
    if(!file) {
      Serial.println("Failed to open file for appending");
      return;
    }
    if(file.println(message)) {
      Serial.println("Data appended");
    } else {
      Serial.println("Append failed");
    }
    file.close();
  }  

  delay(TIME_TO_WAIT);
}

Updates

2020/11/03 Publication of the article

Version française

Click to rate this post!
[Total: 0 Average: 0]

Are you having a problem with this topic?

Maybe someone has already found the solution, visit the forum before asking your question
Ask your question

Share on facebook
Share on twitter
Share on linkedin
Share on pinterest
Share on email
Share on whatsapp

Did you like this project ? Don't miss any more projects by subscribing to our weekly newsletter!

We will be happy to hear your thoughts

      Leave a Reply

      DIY Projects