A while back, I sent this twitter message into the Twitter-verse, with the intention to blog shortly after that on how I got UI5 running on an ESP8266. I'm afraid this blog was conceived a bit later than intended, but here it is nevertheless. This blog is about squeezing a UI5 web application, including the UI5 framework into the ROM of an ESP8266 without adding any additional storage. The available memory space for the UI5 app is 3MB, while the UI5 framework alone is actually many times larger than that. But first a little bit on why I wanted this.
While I was working on my veranda control box, I was also looking for a way to control it remotely. Of course it would be linked up to my OpenHab home automation system using MQTT, but it's always nice to be able to operate it from a web-page as well. This comes in handy when the MQTT server is not properly setup yet or when the ESP8266 chip can't reach the MQTT server for another reason.
It would be easy to build a little UI5 application and get it to run from the ESP8266 while loading the framework from the usual CDN. The files are all static, and the browser takes care of the heavy-lifting when it comes down to a pretty user-interface. The only thing the ESP8266 needs to do is serve some static UI5 application files and serve some JSON/RESTful services that allow the UI5 application to read and change sensors and relays as well as the configuration.
It is getting a bit more complicated when the ESP8266 has just been switched on for the first time and doesn't know what WiFi network to connect to yet. When my veranda control box can't connect to the WiFi network, it will switch itself to access point mode. Once it is in access point mode, you could e.g. take out your phone, connect to this access point and run the setup in the browser to reconfigure it. However, when you're connected to WiFi access point of the ESP8266, you won't have access to the internet at the same time, and you won't be able to load the UI5 framework from a CDN. This means that the entire UI5 framework needs to be hosted on the chip as well.
That's nice: as the entire UI5 framework is way too big for the ESP8266, there's a challenge! The D1 Mini module has an ESP8266 ESP12 onboard, with has 4MB of flash memory. But not all of it can be used for the UI5 application, as it also needs to host a webserver, MQTT client, Thingspeak client, OTA firmware upgrade, sensor and relay logic, etc. So I though a 3MB / 1MB split would be a fair split: 3MB as SPIFFS filesystem for the UI5 application plus framework and 1MB for the firmware. This translated into PlatformIO's `esp8266.flash.4m.ld` layout. If the filesystem available for the UI5 app is going to be 3MB, this would mean that I would have to bring the size of the UI5 application and framework down to less than 3MB!
Performance-wise, it would be a great idea to leverage the '-preload' files as much as possible. It's reduces overhead in the browser, but also from the ESP8266's perspective, it's easier to continue streaming an already opened file than to open another file, and another, and another. Since the preload files contain all modules that a library consists of, leveraging the preload files would be preferred. So let's see what the damage would be in terms of memory usage:
From these files, I would need the libraries sap.m (1.5M), sap.ui.core (1.1M) and sap.ui.commons (576K), I would need the icons font SAP-icons.ttf (264K), the UI5's core sap-ui-core.js (552K) and some jQuery files (20K), totalling up to approximately 4MB. So, even if I would strip the framework down to the bare minimum, it would already exceed the size of the available ROM, and I haven't even added a theme and my application!
To make the framework smaller, it would be necessary to remove modules from the preload files that are not needed by the application. This sounds more daunting than it is, because the library preload files are actually not to difficult to understand. And if you understand how they work, they're also not too difficult to reconstruct, to include only relevant modules. The preload file consists of essentially all of the modules which make up the library. These modules are all minified/compressed and inserted into one file with some metadata wrapped around them. The screen-shot below is an example of such file:
In the screen-shot, you can see that the file starts with some metadata about the library, but quite quickly starts listing the modules that make up this library. In the screenshot, you can see modules `sap.m.AccButton`, `sap.m.AccButtonRenderer`, `sap.m.ActionListItem` and `sap.m.ActionListItemRenderer`. You can even see that there's some space for improvement if the copyright notices for all modules are removed 
So, great! Now the only thing I needed to do was to figure out which modules my application uses to be able to reconstruct the library files with only those modules required by my application. The UI5 framework actually offers a very nice feature to figure this out. If you press Ctrl-Alt-Shift-P, a dialog is displayed that gives some insights in the application. Part of this insight is a list of modules that is currently loaded. So if you launch this dialog after running through the functionality of your application, you'll have a list of all module required to run the application:
I'm interested in the modules that need to be loaded, but I also need to have an overview of what other files I need to get my application to run such as json and css files. This is where the Chrome developer tools come in very handy. After browsing around a little bit, while having the network tab open, I now have a nice collection of all the files that need to be included in the ESP8266's ROM. To be able to automate processing of the contents of the network tab, I have saved the contents as a HAR file, and use tools such as grep to turn it into a packaging script.
Once, you have the HAR file, is is quite easy to make a list of all the files required to run the application using command line tools such as grep, cut and sed:
The eventual script I generated using the contents of the har file and the contents of UI5's debug dialog can be found here:
https://github.com/jpenninkhof/espardu_devicex/blob/master/build-data.sh
In bash this script you can see that almost all files, from both the application and library will be mangled by minifiers. I'm currently using Node.js module 'uglify' for javascript and css files, 'pretty' for xml and json files and 'html-minifier' for html files. After minifying the files, I'm trying to package them up as much as possible into preload files. UI5 framework files go into library preload files, while as many of the application files as possible go into a component preload file.
The total size of the application, including the UI5 framework is now...
A total of 2.2M, which even provides ample room for expansion of the application!
If you're interested in what the app currently looks like, have a look at the little demo below:
Initially the application will show a dashboard, presenting the relay status, temperature and humidity. When you click on the configuration tile, it will be possible to setup the connection to the WiFi access point, MQTT server and Thingspeak. Clicking on the diagnostics page will bring you to the diagnostics dashboard, showing some internal variable of the ESP8266 such as uptime, voltage and available memory. The time in the right-top corner is the current time, which is quite a challenge on an ESP8266 as it doesn't have a real-time clock built-in (and please don't ask me why the time shows a time after mid-night ;). The icon in the top right indicates whether the web-application is currently connected to the ESP8266. After reconfiguration of the ESP8266, it will reboot and the connection icon will shorty show a disconnected state.
As with any OpenUI5 app, the application is fully responsive and plays nicely on Desktop, tablets and phones. UI5 is awesome!
The entire firmware, including the app and build script can be found on Github: https://github.com/jpenninkhof/espardu_devicex. The UI5 app is in the webapp directory.