Datalogging With pbLua
Home / tutorial /
This tutorial describes how to use pbLua to do datalogging. You would use a datalog to collect data from sensors on your robot for later analysis. You can also use the collected data for calibration curves. We'll even show you how to automatically start or select a datalogging application without connecting console when you start the NXT. That might come in handy if you're running the NXT where it's impractical or even impossible to connect a console...
Because we'll be doing a lot of work with files, it would be a really good idea to review the pbLua FLASH File API to make sure you understand all of the features available for datalogging.
We'll also be using the standard Lua string API, which has a lot of functions to deal with building a string out of individual bytes, and packing strings with binary data such as floating point numbers.
As a reminder, there is an excellent online version of Programming in Lua that is a complete description of Lua along with many programming examples, and an even better idea is to purchase the latest Programming in Lua book .
By the end of this tutorial, you'll be able to use the FLASH file system in lots of different applications that require high speed permanent data storage.
Before we store data in files, it would be a good idea to learn how to pack data efficiently in strings. Let's say we want to store readings from an analog sensor like the HiTechnic Gyro.
Besides the raw analog data, it would be a good idea to store the timestamp along with each sensor reading. Here's a simple example program that reads analog data from the gyro and returns it along with a timestamp until the orange button on the NXT is pressed:
-- Read gyro sensor until the orange button is pressed
function GyroRead(port)
-- Set up the gyro sensor with low sensitivity
nxt.InputSetType(port,0)
nxt.InputSetState(port,0,0)
nxt.InputSetDir(port,1,1)
-- Now start reading the sensor and putting out data
repeat
print( nxt.TimerRead(), nxt.InputGetStatus(port) )
until( 8 == nxt.ButtonRead() )
end
-- And using the function - press the orange button on the NXT to stop it
GyroRead(1)
And some typical results for a stationary gyro sensor that has had a bit of time to warm up:
52017 617 0 0 52019 615 0 0 52021 617 0 0 52023 616 0 0 52025 616 0 0 52027 617 0 0 52029 616 0 0
The values in each line correspond to:
The analog values have up to 10 bits of resolution, so they can range from 0 to 1023. The value of 616 is typical for the gyro at rest.
If you're interested, you can count the characters, but it looks like if we store just the timestamp and the raw analog data it will take about 11 bytes to store the string. If you were to run this program later, it might take even more bytes to hold the string depending on the timestamp.
Let's assume a worst case of 16 bytes per line of data. If we allocate 16K to a data file, that's only 1,000 readings that we can store. We can actually allocate up to 64K for a single pbLua data file, in case you're curious.
If we take a reading 10 times a second on a really interesting ride, like a roller coaster, that's about 100 seconds of data in 16K.
We're going to have to find some ways to compact our raw data strings, and fortunately it's easy and quick to do this with pbLua.
The standard Lua string API does not provide a way to convert a full sized 32 bit number to its equivalent individual bytes. The pbLua API, however, has these functions:
What do we mean by native byte order? If you were to take the hex value 0x12345678 and convert it, the result would be 0x78563412. That's because the ARM7 inside the NXT is a "little-endian" processor and we read numbers using "big endian" notation. For more information on this subject, read this article on endianness.
OK, so getting back on track, it's clear we can easily convert an integer or a float to a series of bytes. To save our gyro reading and timestamp, it's as easy as this:
-- Lots of tricks in this function, note the concatenation operator ".." and -- that the last two results from nxt.InputGetStatus() are simply discarded! function sampleString(port) return nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(port)) end
And if you follow along with the next results, you'll see more examples of what the native little-endian numbers look like, and how easy it is to convert between numbers and byte strings.
-- Let's do a few examples first... > =string.byte(nxt.istr(0),1,4) 0 0 0 0 > =string.byte(nxt.istr(255),1,4) 255 0 0 0 > =string.byte(nxt.istr(65535),1,4) 255 255 0 0 > =string.byte(nxt.istr(-1),1,4) 255 255 255 255 -- And now use the function to generate the string... > port = 1 > s = sampleString(port) -- And dump the results. Can you figure out what the values were? > =string.byte(s,1,8) 148 26 10 0 106 2 0 0 -- It's easy if you do this... > -- The timestamp is the first 4 bytes (1 to 4) > =nxt.stri(string.sub(s,1,4)) 662164 -- The value is the second 4 bytes (5 to 8) > =nxt.stri(string.sub(s,5,8)) 618
Now that we know how to compact numnbers into raw bytes, and how to get the raw numbers back again later, we can move on to creating datalogs in the pbLua Flash File System.
If you have not yet looked over the pbLua FLASH File API then it might be a good idea to do that now. To store data in a file with pbLua we'll need the following information:
Recall that the pbLua Flash File System is not quite the same as to one on your computer. It's simple minded, and needs to know how much data you'll be trying to store so that it can find a block of unused FLASH big enough for your file. It's better to find out that there's not enough space beofre you start writing data.
Remember that you'll want a unique name for your data file. If you want to use the same file name every time, then you'll need to erase the old file before you start.
Just for fun, let's write a little function that reads data from an analog port and writes it as fast as possible. If we timestamp every write, we'll have an idea of how quickly we can write data to the filesysytem. If we save a timestamp (4 bytes) and data (4 bytes) we'll need 8 bytes times 1000 records or 8000 bytes in our file.
-- Function to save 100 timestamped samples to a file as quickly as
-- possible
function save100 (port)
-- Set up the gyro sensor with low sensitivity
nxt.InputSetType(port,0)
nxt.InputSetState(port,0,0)
nxt.InputSetDir(port,1,1)
-- Create an 800 byte file, save the handle
local file = nxt.FileCreate("sampleFile", 800)
-- And now read and save 100 individual samples and timestamps
for i=1,100 do
nxt.FileWrite( file, nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(port)) )
end
-- And close the file
nxt.FileClose(file)
end
-- And run the function on port 1 to try it out...
save100(1)
Of course, now that you've written all the data to a file, it would be nice to read it back, so we'll use the dumping function in this next example to do it, and I've even pasted the first few and last few records to see how long it took...
-- Function to dump timestamped samples from a file
function DataDumper()
-- Open the file
file = nxt.FileOpen( "sampleFile" )
-- And now read individual samples and timestamps 8 bytes
-- at a time until there are no more
repeat
local s = nxt.FileRead( file, 8 )
if s then
local t = nxt.stri(string.sub(s,1,4))
local v = nxt.stri(string.sub(s,5,8))
print( string.format( "T:%08i V:%04i", t, v ) )
end
until nil == s
-- And close the file
nxt.FileClose(file)
end
-- And run the function to see how fast the writing went...
DataDumper()
T:00950217 V:0617
T:00950221 V:0616
T:00950224 V:0619
...
T:00950574 V:0616
T:00950578 V:0615
T:00950581 V:0618
-- Yep, that's 100 samples written in 364 msec!
WOW! That's reading 100 timestamps and cycles and writing the results in 364 msec - almost 300 Hz!
If you've read some of the other tutorials, you'll know that the last section is usually the place where we put together a bunch of stuff we've learned in the previous section. This is no different. We'll modify the basic datalogger to handle the case of erasing the file if it already exists. We'll also let it use 8,000 bytes for 1000 entries, and finally, we'll log every 100 msec or 10 times per second. This gives us about 100 seconds of data.
We'll add a few other nice touches, such as a real-time update of progress on the LCD and the ability to stop logging and turn off the NXT when the user presses the orange button or when the end of the file is reached.
Here's what the new program looks like:
-- Complete data logger example
s = [[
function DataLogger ()
-- Set up the gyro sensor on port 1 with low sensitivity
nxt.InputSetType(1,0)
nxt.InputSetState(1,0,0)
nxt.InputSetDir(1,1,1)
-- Check to see if the file exists, and if so, erase it...
if nxt.FileExists("sampleFile") then
nxt.FileDelete("sampleFile")
end
-- Create an 8000 byte file, save the handle
local file = nxt.FileCreate("sampleFile", 8000)
-- Clear the display
nxt.DisplayClear()
-- And now read and save up to 100 individual samples and timestamps
for i=1,1000 do
nxt.FileWrite( file, nxt.istr(nxt.TimerRead()) .. nxt.istr(nxt.InputGetStatus(1)) )
-- Update the LCD so we can see what's going on...
nxt.DisplayText( i )
-- break out of the loop if the user hits the orange button
if 8 == nxt.ButtonRead() then
break
end
end
-- And close the file
nxt.FileClose(file)
-- And turn off the NXT
nxt.PowerDown()
end
-- Don't forget tot execute it :-)
DataLogger()
]]
What's going on here? What's with the text inside the "[[" "]]" characters? It's called a literal string, and you can read more about them here. We'll be using this string to load up our file with the actual code.
WARNING: Always, always,always check and recheck your code before comitting it to FLASH. Then check it again. Especially if you're expecting the code to work when the NXT is started automatically. You can always override things if you find a bug by connecting to the NXT with a USB cable. It will drop you to the console where you can fix your problem.
So, we've got the function we want to put into FLASH into a string called s. First, figure out how long the string is:
> =string.len(s) 970
To be safe, we'll let the file system know that we want to store 1,024 bytes, so let's make the startup file. Remember to erase the old one if you already have one.
> f=nxt.FileCreate("pbLuaStartup", 1024)
> nxt.FileWrite(f,s)
> nxt.FileClose(f)
That's all there is to it! When you cycle power to the NXT, you'll see that the datalogger will start automatically!
Now, before you get all excited, there is a big issue lurking in the Datalogger() code. Think about it for a while before you read the next paragraph...
The main problem I see is that if you have erased lots of files, you may run out of file descriptors when creating the sampleFile. To avoid this, make sure you do a FileFormat(0) before you commit your NXT to an expensive experiment. Make sure you don't do a FileFormat(1) or you'll erase all of the files, including the startup file.
OK, now go out and do some datalogging!
by Roberto Ierusalimschy
by Jon Bentley