Reducing the Memory Usage of pbLua
Home / tutorial /
This tutorial describes how to reduce the RAM usage of pbLua for versions starting at Beta 18a. We'll discuss how to "delete" function tables and how to load only the parts you will actually use.
The standard Lua require() builtin function will be used in the pbLua application, but it works a bit differently.
As a reminder, there is an excellent online version of Programming in Lua that is a complete description of Lua along with many programming examples, and an even better idea is to purchase the latest Programming in Lua book .
By the end of this tutorial, you'll be stripping down your Lua system and only loading the parts you really need.
The pbLua application code lives in the FLASH memory of the Atmel AT91SAM7 microcontroller in the NXT. There is 256K of FLASH memory available, and the pbLua image uses about 132K of that. This leaves about 124K for pbLua scripts and datalogs.
The AT91SAM7 device also has 64K of RAM available for the application, and this is the resource that's in short supply. Before pbLua even starts running, there is about 4K of RAM used for assorted internal variables, buffers, etc to allow the various drivers to operate. There is also about 3.5K allocated to the application stack, which leaves about 58.5K of RAM for use by the application.
pbLua is an interpreted language. It reads text from the console (or a file) and compiles it into intermediate bytecodes, which are then executed by a very efficient bytecode interpreter. In the course of compiling the text and executing the resulting code, pbLua makes use of dynamic memory. This is memory that is continuously allocated, resized, and freed from the free memory heap. Refer to this article on memory managers for more details.
Once pbLua has initialized, it uses about 16K of memory. You can always determine how much memory pbLua thinks it is using by using one of the functions in the example below:
-- gcinfo() returns the approximate memory use in K (1024 byte) multiples
>=gcinfo()
16
-- This tells us we are using about 16*1024 = 16,384 bytes
--
-- For a more detailed estimate, use collectgarbage(), like this
-- The first value is the approximate memory use in K (1024 byte) multiples
-- and the second value is the fractional K values (mod 1024)
>=collectgarbage("count")
16 403
-- This tells us we are using about 16*1024 + 403 = 16,787 bytes
--
--Use the second form when you need a more accurate estimate of RAM usage
Of course, this is not the entire story. The memory estimate from the pbLua application is what it thinks it's using, as if the memory is all in one big chunk. In fact, pbLua allocates a lot of little chunks of memory, using a memory allocator. Memory allocators suffer from a problem called fragmentation - it's basically the space "in between" the little chunks of memory that are too small to be used for anything. A typical memory allocator will suffer between 20 and 30 percent fragmentation, unless you get really sophisticated.
There's a little helper routine that dumps your heap information and lets you get a picture of the memory fragmentation:
-- Basic heap information available from pbLua >=nxt.HeapInfo() 454 443 11 6728 2434 4294 -- This tells us two important things: -- -- We have 454 "chunks" of memory in our heap, 443 are in use, and 11 are free -- -- We have 6728 "blocks" of memory (each 8 bytes), 2434 are in use, 4294 are free
If you want an even more detailed picture of the current heap usage, you can dump the entire chain like this:
>nxt.HeapInfo(1) Dumping the umm_heap... |0x00201fb8|B 0|NB 1|PB 0|Z 1|NF 3096|PF 0| |0x00201fc0|B 1|NB 43|PB 0|Z 42| |0x00202110|B 43|NB 68|PB 1|Z 25| ... |0x00208118|B 3116|NB 3120|PB 3111|Z 4| |0x00208138|B 3120|NB 3124|PB 3116|Z 4| |0x00208158|B 3124|NB 3130|PB 3120|Z 6|NF 3148|PF 3096| |0x00208188|B 3130|NB 3133|PB 3124|Z 3| |0x002081a0|B 3133|NB 3144|PB 3130|Z 11| |0x002081f8|B 3144|NB 3148|PB 3133|Z 4| |0x00208218|B 3148|NB 3149|PB 3144|Z 1|NF 3070|PF 3124| |0x00208220|B 3149|NB 0|PB 3148|Z 3580|NF 0|PF 136| Total Entries 607 Used Entries 563 Free Entries 44 Total Blocks 6728 Used Blocks 3072 Free Blocks 3656 -- This somewhat truncated wiew of the heap dump lets you figure out exactly -- how fragmented the heap is - if you are so inclined
If you're read any of the Lua programming resources that I recommend, then you'll know that everything in Lua is a table. That goes for function libraries too. Let's dump the debug library table to see what's inside it:
-- Dump the contents of the "debug" table using the pairs() iterator, the -- following output is slightly altered for readability... >for k,v in pairs(debug) do print(k,v) end getupvalue function:0x204214 debug function:0x2040ec sethook function:0x204264 getmetatable function:0x2041fc gethook function:0x20411c setmetatable function:0x2042d4 setlocal function:0x20429c traceback function:0x204324 setfenv function:0x20424c getinfo function:0x204154 setupvalue function:0x2042ec getlocal function:0x20418c getregistry function:0x2041c4 getfenv function:0x204104
The debug library table has nothing but functions in it, but there could be strings, more tables, values, you name it. Recall that the variable debug is just a data structure that holds the table. We can set the variable to whatever we want, and as long as the old contents are not referred to anywhere else, they are eligible for garbage collection.
We'll use the convention of setting the table variable to the special value of nil which makes both the variable and the table it used to hold eligible for garbage collection. This next example shows the space savings we can get from eliminating the debug library table:
-- This example shows the RAM savings we get by eliminating the debug table
--
-- First, do a round of garbage collection...
=collectgarbage()
-- Now print out the current memory usage...
=collectgarbage("count")
16 776
-- That's (16*1024)+776 = 17160 bytes in use
--
-- Now let's delete the debug table, do a round of garbage collection, and
-- print the new memory usage...
debug=nil
=collectgarbage()
-- Now print out the current memory usage...
=collectgarbage("count")
15 938
-- That's (15*1024)+939 = 16299 bytes in use
As a final exercise, we'll strip down the table, and the nxt libraries and see how much memory we save...
-- Now delete the table, string, and nxt libraries and see how much RAM is
-- in use...
table=nil
string=nil
nxt=nil
=collectgarbage()
-- Now print out the current memory usage...
=collectgarbage("count")
7 698
-- That's (7*1024)+698 = 7866 bytes in use - a pretty decent savings!
Not bad. We've got down to a little over 7.5K of RAM in use. This is of little use to us now though, because there are no table or nxt functions, and we can't even display anything on the NXT LCD! The next section will get back the tables we've removed, but this time we'll only load the tables we actually need.
At this point, we've got a stripped down pbLua system that doesn't do very much. So how do we get back the functions we've lost? That's where the require() function comes in.
The first thing that we'll discover about require is that if you pass no parameters, it returns a list of all the tables that it knows how to get back, like this:
-- What does require() return? >=require() table:0x204c74 -- It's a table, so let's put the table through an iterator to what's in it... >for k,v in pairs(require()) do print(k,v) end 1 all 2 table 3 string 4 debug 5 nxt_misc 6 nxt_bt 7 nxt_math 8 nxt_file 9 nxt_output 10 nxt_display 11 nxt_i2c 12 nxt_rs485 13 nxt_input 14 nxt_sound
Hmmm, that looks useful. It's a list of all the libraries that require() knows about! You probably know the next step. Calling require() with the name of the library (as a string) will load it up for us. Let's see if we can get the debug library back.
-- First, let's dump debug - it should give an error since it's gone...
for k,v in pairs(debug) do print(k,v) end
tty: stdin:1: bad argument #1 to 'pairs' (table expected, got nil)
-- Now let's load up the debug library and dump it. I've reformatted the
-- output slightly...
require("debug")
for k,v in pairs(debug) do print(k,v) end
getupvalue function:0x204ccc
debug function:0x203274
sethook function:0x204c7c
getmetatable function:0x204ce4
gethook function:0x203734
setmetatable function:0x20315c
setlocal function:0x204c44
traceback function:0x20310c
setfenv function:0x204c94
getinfo function:0x2036fc
setupvalue function:0x203144
getlocal function:0x204d54
getregistry function:0x204d1c
getfenv function:0x20374c
That's pretty easy. You should be able to load up any of the other tables to get the exact subset of functions that you want, like nxt_display or nxt_output.
Finally, if you want to get everything back the way it was, which means loading all the tables at once, just use require("all").
There's really not much to managing the RAM usage of the pbLua libraries. To delete a library, just set its table variable to nil. To get the library back, just call require() with the library name.
by Roberto Ierusalimschy
by Jon Bentley