esp8266 firmware and filesystem update with FTP client - 2 (original) (raw)

In this series of article about firmware and how to update It, I’d like to add a series of alternative methods of updating that are very useful for me.

esp8266 firmware and filesystem update with FTP client

In this article, we will learn how to add an FTP server to our device and use It to upload firmware and filesystem updates.

Toggle

• Update firmware from a storage with Update class
• Upload firmware via FTP and automatic update
  • Update firmware with FTP server over LittleFS
• Update filesystem with FTP server over an SD
• Thanks

Update firmware from a storage with Update class

The basic methods to update a stream of data are contained in the Update class.

A method, in particular, can manage updates via stream

Update.begin(firmwareSizeInBytes); Update.writeStream(streamVar); Update.end();

Upload firmware via FTP and automatic update

Another very interesting system to update firmware is by using an FTP client.

I have created some applications for storing logs and statistics and then downloading the data via an FTP client. To do that, I use a simple library that you can check on this article “FTP server on esp8266 and esp32” I created that library, and I customized It with some callback, and you can use them for our purpose.

We can use SPIFFS LittleFS or SD. Check on these articles the basics:

• WeMos D1 mini (esp8266), integrated SPIFFS Filesystem (deprecated from core 3.0)
• WeMos D1 mini (esp8266), integrated LittleFS Filesystem
• How to use SD card with esp8266 and Arduino

Update firmware with FTP server over LittleFS

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 #define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 NETWORK_ESP8266 #define DEFAULT_STORAGE_TYPE_ESP8266 STORAGE_LITTLEFS #endif

To understand how to generate the compiled firmware, read this article “esp8266: flash firmware binary (.bin) compiled and signed“, which is quite simple but is mandatory to understand all the steps.

Here I write an example that uses the SimpleFTPServer library to upload the firmware.bin file on LittleFS, and when the system store that file, it starts the update of firmware.

/*

• Upload firmware with FtpServer (esp8266 with LittleFS)
• when uploaded start automatic Update and reboot
• AUTHOR: Renzo Mischianti
• https://mischianti.org/
• */

#include <ESP8266WiFi.h> #include <LittleFS.h>

#include <SimpleFTPServer.h>

const char* ssid = ""; const char* password = "";

FtpServer ftpSrv; //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial

bool isFirmwareUploaded = false;

void progressCallBack(size_t currSize, size_t totalSize) { Serial.printf("CALLBACK: Update process at %d of %d bytes...\n", currSize, totalSize); }

#define FIRMWARE_VERSION 0.2

void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){ switch (ftpOperation) { case FTP_CONNECT: Serial.println(F("FTP: Connected!")); break; case FTP_DISCONNECT: Serial.println(F("FTP: Disconnected!")); break; case FTP_FREE_SPACE_CHANGE: if (isFirmwareUploaded){ Serial.println(F("The uploaded firmware now stored in FS!")); Serial.print(F("\nSearch for firmware in FS..")); String name = "firmware.bin"; File firmware = LittleFS.open(name, FTP_FILE_READ); if (firmware) { Serial.println(F("found!")); Serial.println(F("Try to update!"));

        Update.onProgress(progressCallBack);

        Update.begin(firmware.size(), U_FLASH);
        Update.writeStream(firmware);
        if (Update.end()){
        Serial.println(F("Update finished!"));
        }else{
        Serial.println(F("Update error!"));
        Serial.println(Update.getError());
        }

        firmware.close();

        String renamed = name;
        renamed.replace(".bin", ".bak");
        if (LittleFS.rename(name, renamed.c_str())){
        Serial.println(F("Firmware rename succesfully!"));
        }else{
        Serial.println(F("Firmware rename error!"));
        }
        delay(2000);

        ESP.reset();
      }else{
        Serial.println(F("not found!"));
      }
      // isFirmwareUploaded = false; // not need by reset
  }
  Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
  break;
default:
  break;

} }; void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){ switch (ftpOperation) { case FTP_UPLOAD_START: Serial.println(F("FTP: Upload start!")); break; case FTP_UPLOAD: Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize); break; case FTP_TRANSFER_STOP: Serial.println(F("FTP: Finish transfer!")); break; case FTP_TRANSFER_ERROR: Serial.println(F("FTP: Transfer error!")); break; default: break; }

/* FTP_UPLOAD_START = 0,

• FTP_UPLOAD = 1,
• FTP_DOWNLOAD_START = 2,
• FTP_DOWNLOAD = 3,
• FTP_TRANSFER_STOP = 4,
• FTP_DOWNLOAD_STOP = 4,
• FTP_UPLOAD_STOP = 4,
• FTP_TRANSFER_ERROR = 5,
• FTP_DOWNLOAD_ERROR = 5,
• FTP_UPLOAD_ERROR = 5 */

if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){ isFirmwareUploaded = true; } };

void setup(void){ Serial.begin(115200); WiFi.begin(ssid, password); Serial.println("");

// Wait for connection while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.print("Connected to "); Serial.println(ssid); Serial.print("IP address: "); Serial.println(WiFi.localIP());

ftpSrv.setCallback(_callback); ftpSrv.setTransferCallback(_transferCallback);

/////FTP Setup, ensure LittleFS is started before ftp; ///////// if (LittleFS.begin()) { Serial.println("LittleFS opened!"); Serial.print(F("\nCurrent firmware version: ")); Serial.println(FIRMWARE_VERSION);

  ftpSrv.begin("esp8266","esp8266");    //username, password for ftp.   (default 21, 50009 for PASV)

} } void loop(void){ ftpSrv.handleFTP(); //make sure in loop you call handleFTP()!! // server.handleClient(); //example if running a webserver you still need to call .handleClient();

}

The code is not so complicated, but you must understand only a few steps.

When FTP uploads the file in the callback, you can distinguish three phases: the FTP_UPLOAD_START, FTP_UPLOAD, and FTP_TRANSFER_STOP.

When transfer stops, the file is completely transferred

if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){ isFirmwareUploaded = true; }

but not yet stored, and you must check the event FTP_FREE_SPACE_CHANGE to ensure that the file is ultimately stored.

switch (ftpOperation) { case FTP_CONNECT: Serial.println(F("FTP: Connected!")); break; case FTP_DISCONNECT: Serial.println(F("FTP: Disconnected!")); break; case FTP_FREE_SPACE_CHANGE: if (isFirmwareUploaded){ Serial.println(F("The uploaded firmware now stored in FS!")); Serial.print(F("\nSearch for firmware in FS..")); String name = "firmware.bin"; File firmware = LittleFS.open(name, FTP_FILE_READ); if (firmware) { Serial.println(F("found!")); Serial.println(F("Try to update!"));

        Update.onProgress(progressCallBack);

        Update.begin(firmware.size(), U_FLASH);
        Update.writeStream(firmware);
        if (Update.end()){
        Serial.println(F("Update finished!"));
        }else{
        Serial.println(F("Update error!"));
        Serial.println(Update.getError());
        }

        firmware.close();

        String renamed = name;
        renamed.replace(".bin", ".bak");
        if (LittleFS.rename(name, renamed.c_str())){
        Serial.println(F("Firmware rename succesfully!"));
        }else{
        Serial.println(F("Firmware rename error!"));
        }
        delay(2000);

        ESP.reset();
      }else{
        Serial.println(F("not found!"));
      }
      // isFirmwareUploaded = false; // not need by reset
  }
  Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
  break;
default:
  break;

}

Now I check if the file is present and if yes, the update starts the stream of the file; after upload, It is renamed and start the reboot that does the final phase of the update.

....... Connected to reef-casa-sopra IP address: 192.168.1.127 LittleFS opened!

Current firmware version: 0.10 FTP: Connected! FTP: Upload start! FTP: Upload of file firmware.bin byte 2048 FTP: Upload of file firmware.bin byte 3216 FTP: Upload of file firmware.bin byte 3752 FTP: Upload of file firmware.bin byte 5360 FTP: Upload of file firmware.bin byte 7408 FTP: Upload of file firmware.bin byte 8040 FTP: Upload of file firmware.bin byte 9112 FTP: Upload of file firmware.bin byte 11160 FTP: Upload of file firmware.bin byte 11792 FTP: Upload of file firmware.bin byte 12864 FTP: Upload of file firmware.bin byte 14912 FTP: Upload of file firmware.bin byte 15544 FTP: Upload of file firmware.bin byte 16080 FTP: Upload of file firmware.bin byte 17688

[...]

FTP: Upload of file firmware.bin byte 327400 FTP: Upload of file firmware.bin byte 329104 FTP: Upload of file firmware.bin byte 329640 FTP: Upload of file firmware.bin byte 330176 FTP: Upload of file firmware.bin byte 331784 FTP: Upload of file firmware.bin byte 333832 FTP: Upload of file firmware.bin byte 335536 FTP: Upload of file firmware.bin byte 336072 FTP: Upload of file firmware.bin byte 338120 FTP: Upload of file firmware.bin byte 338752 FTP: Upload of file firmware.bin byte 340360 FTP: Upload of file firmware.bin byte 341600 FTP: Finish transfer! The uploaded firmware now stored in FS!

Search for firmware in FS..found! Try to update! CALLBACK: Update process at 0 of 341600 bytes... CALLBACK: Update process at 4096 of 341600 bytes... CALLBACK: Update process at 8192 of 341600 bytes... CALLBACK: Update process at 12288 of 341600 bytes... CALLBACK: Update process at 16384 of 341600 bytes... CALLBACK: Update process at 20480 of 341600 bytes... CALLBACK: Update process at 24576 of 341600 bytes... CALLBACK: Update process at 28672 of 341600 bytes... CALLBACK: Update process at 32768 of 341600 bytes... CALLBACK: Update process at 36864 of 341600 bytes...

[...]

CALLBACK: Update process at 319488 of 341600 bytes... CALLBACK: Update process at 323584 of 341600 bytes... CALLBACK: Update process at 327680 of 341600 bytes... CALLBACK: Update process at 331776 of 341600 bytes... CALLBACK: Update process at 335872 of 341600 bytes... CALLBACK: Update process at 339968 of 341600 bytes... CALLBACK: Update process at 341600 of 341600 bytes... CALLBACK: Update process at 341600 of 341600 bytes... Update finished! Firmware rename succesfully!

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 tail 4 chksum 0xcc load 0x3fff20b8, len 40, room 4 tail 4 chksum 0xc9 csum 0xc9 v00053100 @cp:B0 ld

....... Connected to reef-casa-sopra IP address: 192.168.1.127 LittleFS opened!

Current firmware version: 0.20

Update filesystem with FTP server over an SD

For this test, remember to modify (or check) that FtpServerKey.h is like so

#ifndef DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 #define DEFAULT_FTP_SERVER_NETWORK_TYPE_ESP8266 NETWORK_ESP8266 #define DEFAULT_STORAGE_TYPE_ESP8266 STORAGE_SD #endif

This sketch isn’t more complex, I reuse a part of previous code, and I add the management of the filesystem.bin file.

We create a version.txt file on the sketch’s data folder, and I write 0.1 inside, and I use that as the version of the FileSystem, then upload the data folder to the filesystem.

Now we are going to modify the file version.txt with the version 0.2 then upload to the device, then

Change version to 0.2 in version.txt, regenerate without upload, and rename the file ArduinoOTAesp8266_fs_update.mklittlefs.bin to filesystem.bin and upload the file to the SD.

Here is the complete sketch.

/*

• Upload firmware or filesystem (LittleFS) with FtpServer (esp8266 with SD)
• when uploaded start automatic Update and reboot
• AUTHOR: Renzo Mischianti
• https://mischianti.org/
• */

#include <ESP8266WiFi.h> #include <SD.h> #include <LittleFS.h>

#include <SimpleFTPServer.h>

const char* ssid = ""; const char* password = "";

FtpServer ftpSrv; //set #define FTP_DEBUG in ESP8266FtpServer.h to see ftp verbose on serial

bool isFirmwareUploaded = false; bool isFilesystemUploaded = false;

void progressCallBack(size_t currSize, size_t totalSize) { Serial.printf("CALLBACK: Update process at %d of %d bytes...\n", currSize, totalSize); }

#define FIRMWARE_VERSION 0.2 String FILESYSTEM_VERSION = "0.0";

void _callback(FtpOperation ftpOperation, unsigned int freeSpace, unsigned int totalSpace){ switch (ftpOperation) { case FTP_CONNECT: Serial.println(F("FTP: Connected!")); break; case FTP_DISCONNECT: Serial.println(F("FTP: Disconnected!")); break; case FTP_FREE_SPACE_CHANGE: if (isFirmwareUploaded){ Serial.println(F("The uploaded firmware now stored in FS!")); Serial.print(F("\nSearch for firmware in FS..")); String name = "firmware.bin"; File firmware = SD.open(name, FTP_FILE_READ); if (firmware) { Serial.println(F("found!")); Serial.println(F("Try to update!"));

        Update.onProgress(progressCallBack);

        Update.begin(firmware.size(), U_FLASH);
        Update.writeStream(firmware);
        if (Update.end()){
        Serial.println(F("Update finished!"));
        }else{
        Serial.println(F("Update error!"));
        Serial.println(Update.getError());
        }

        firmware.close();

        String renamed = name;
        renamed.replace(".bin", ".bak");
        if (SD.rename(name, renamed.c_str())){
        Serial.println(F("Firmware rename succesfully!"));
        }else{
        Serial.println(F("Firmware rename error!"));
        }
        delay(2000);

        ESP.reset();
      }else{
        Serial.println(F("not found!"));
      }
      // isFirmwareUploaded = false; // not need by reset
  }
  if (isFilesystemUploaded){
      Serial.println(F("The uploaded Filesystem now stored in FS!"));
      Serial.print(F("\nSearch for Filesystem in FS.."));
      String name = "filesystem.bin";
      File filesystem =  SD.open(name, FTP_FILE_READ);
      if (filesystem) {
        Serial.println(F("found!"));
        Serial.println(F("Try to update!"));

        Update.onProgress(progressCallBack);

        Update.begin(filesystem.size(), U_FS);
        Update.writeStream(filesystem);
        if (Update.end()){
        Serial.println(F("Update finished!"));
        }else{
        Serial.println(F("Update error!"));
        Serial.println(Update.getError());
        }

        filesystem.close();

        String renamed = name;
        renamed.replace(".bin", ".bak");
        if (SD.rename(name, renamed.c_str())){
        Serial.println(F("Filesystem rename succesfully!"));
        }else{
        Serial.println(F("Filesystem rename error!"));
        }
        delay(2000);

        ESP.reset();
      }else{
        Serial.println(F("not found!"));
      }
      // isFilesystemUploaded = false; // not need by reset
  }
  Serial.printf("FTP: Free space change, free %u of %u!\n", freeSpace, totalSpace);
  break;
default:
  break;

} }; void _transferCallback(FtpTransferOperation ftpOperation, const char* name, unsigned int transferredSize){ switch (ftpOperation) { case FTP_UPLOAD_START: Serial.println(F("FTP: Upload start!")); break; case FTP_UPLOAD: Serial.printf("FTP: Upload of file %s byte %u\n", name, transferredSize); break; case FTP_TRANSFER_STOP: Serial.println(F("FTP: Finish transfer!")); break; case FTP_TRANSFER_ERROR: Serial.println(F("FTP: Transfer error!")); break; default: break; }

/* FTP_UPLOAD_START = 0,

• FTP_UPLOAD = 1,
• FTP_DOWNLOAD_START = 2,
• FTP_DOWNLOAD = 3,
• FTP_TRANSFER_STOP = 4,
• FTP_DOWNLOAD_STOP = 4,
• FTP_UPLOAD_STOP = 4,
• FTP_TRANSFER_ERROR = 5,
• FTP_DOWNLOAD_ERROR = 5,
• FTP_UPLOAD_ERROR = 5 */

if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("firmware.bin")>=0){ isFirmwareUploaded = true; } if (ftpOperation == FTP_UPLOAD_STOP && String(name).indexOf("filesystem.bin")>=0){ isFilesystemUploaded = true; } };

void setup(void){ Serial.begin(115200); WiFi.begin(ssid, password); Serial.println("");

// Wait for connection while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.print("Connected to "); Serial.println(ssid); Serial.print("IP address: "); Serial.println(WiFi.localIP());

ftpSrv.setCallback(_callback); ftpSrv.setTransferCallback(_transferCallback);

Serial.print(F("Inizializing FS...")); if (LittleFS.begin()){ Serial.println(F("done.")); }else{ Serial.println(F("fail.")); }

Serial.print(F("FileSystem version ")); File versionFile = LittleFS.open(F("/version.txt"), "r"); if (versionFile) { FILESYSTEM_VERSION = versionFile.readString(); versionFile.close(); } Serial.println(FILESYSTEM_VERSION);

/////FTP Setup, ensure SD is started before ftp; ///////// if (SD.begin(SS)) { Serial.println("SD opened!"); Serial.print(F("\nCurrent firmware version: ")); Serial.println(FIRMWARE_VERSION);

  ftpSrv.begin("esp8266","esp8266");    //username, password for ftp.   (default 21, 50009 for PASV)

} } void loop(void){ ftpSrv.handleFTP(); //make sure in loop you call handleFTP()!! // server.handleClient(); //example if running a webserver you still need to call .handleClient();

}

The serial output when I upload the file filesystem.bin

... Connected to reef-casa-sopra IP address: 192.168.1.127 Inizializing FS...done. FileSystem version 0.1 SD opened!

Current firmware version: 0.20 FTP: Connected! FTP: Free space change, free 1 of 1! FTP: Free space change, free 1 of 1! FTP: Free space change, free 1 of 1! FTP: Upload start! FTP: Upload of file filesystem.bin byte 1608 FTP: Upload of file filesystem.bin byte 2680 FTP: Upload of file filesystem.bin byte 3216 FTP: Upload of file filesystem.bin byte 4288 FTP: Upload of file filesystem.bin byte 4824 FTP: Upload of file filesystem.bin byte 5360 FTP: Upload of file filesystem.bin byte 6968 FTP: Upload of file filesystem.bin byte 7504 FTP: Upload of file filesystem.bin byte 8040 FTP: Upload of file filesystem.bin byte 9112 FTP: Upload of file filesystem.bin byte 9648 FTP: Upload of file filesystem.bin byte 10184

[...]

FTP: Upload of file filesystem.bin byte 2068960 FTP: Upload of file filesystem.bin byte 2069496 FTP: Upload of file filesystem.bin byte 2070568 FTP: Upload of file filesystem.bin byte 2071104 FTP: Upload of file filesystem.bin byte 2071640 FTP: Upload of file filesystem.bin byte 2072576 FTP: Finish transfer! The uploaded Filesystem now stored in FS!

Search for Filesystem in FS..found! Try to update! CALLBACK: Update process at 0 of 2072576 bytes... CALLBACK: Update process at 4096 of 2072576 bytes... CALLBACK: Update process at 8192 of 2072576 bytes... CALLBACK: Update process at 12288 of 2072576 bytes... CALLBACK: Update process at 16384 of 2072576 bytes... CALLBACK: Update process at 20480 of 2072576 bytes... CALLBACK: Update process at 24576 of 2072576 bytes...

[...]

CALLBACK: Update process at 2052096 of 2072576 bytes... CALLBACK: Update process at 2056192 of 2072576 bytes... CALLBACK: Update process at 2060288 of 2072576 bytes... CALLBACK: Update process at 2064384 of 2072576 bytes... CALLBACK: Update process at 2068480 of 2072576 bytes... CALLBACK: Update process at 2072576 of 2072576 bytes... CALLBACK: Update process at 2072576 of 2072576 bytes... Update finished! Filesystem rename succesfully!

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3460, room 16 tail 4 chksum 0xcc load 0x3fff20b8, len 40, room 4 tail 4 chksum 0xc9 csum 0xc9 v00070bd0 ~ld

....... Connected to reef-casa-sopra IP address: 192.168.1.127 Inizializing FS...done. FileSystem version 0.2 SD opened!

Current firmware version: 0.20

Thanks

1. Firmware management
   1. esp8266: flash firmware binary (.bin) compiled and signed
   2. esp8266: flash firmware and filesystem binary (.bin) compiled with GUI tools
2. OTA update with Arduino IDE
   1. esp8266 OTA update with Arduino IDE: filesystem, signed and password
3. OTA update with Web Browser
   1. esp8266 OTA update with Web Browser: firmware, filesystem and authentication
   2. esp8266 OTA update with Web Browser: sign the firmware and HTTPS (SSL/TLS)
   3. esp8266 OTA update with Web Browser: custom web interface
4. Self OTA uptate from HTTP server
   1. esp8266 self OTA update firmware from server
   2. esp8266 self OTA update firmware from server with version check
   3. esp8266 self OTA update in HTTPS (SSL/TLS) with trusted self signed certificate
5. Non standard Firmware update
   1. esp8266 firmware and filesystem update from SD card
   2. esp8266 firmware and filesystem update with FTP client