You’ve heard it before: “Premature optimization is the root of all Evil.” Capital Evil. So you go on about your day, arranging the ones and zeros in pretty christmas tree shapes and suddenly the day arrives: your program is slow as molasses. What are you going to do now?
Last monday was that day for me, and I’ve been underwater ever since. “Why is this happening to me?!” I thought. While not prematurely optimizing, I thought I did things right. I have no nested for loops. I’m not using an array when I need a list. Threads aren’t modifying the UI willy-nilly. Why has God forsaken me?
I first noticed it while working on the printer module. After the program is open for some length of time, my whole computer begins to lag. Not just a little bit either; things completely fall apart. In the space of about 5 minutes, the computer becomes unusably slow. Killing the Photo Booth process doesn’t help; only physically shutting the computer off helps. Of course, the computer is so slow that I can’t use the shutdown option; I have to press The Button.
At this point, I feel some context is in order. I had been trying to figure out how to make my printer print on photo paper. Apparently printing is one of the areas Linux still hasn’t caught up to windows on, so this was proving to be difficult. After printing a few strips, I realized that my low-res photo strips weren’t going to cut it, so I bumped the resolution from 100 pixels wide to 1000. It was then that I noticed things were off.
Ten years of troubleshooting experience kicked in: “what changed?” I thought. The obvious answer was the image size. Clearly my photo strip assembly algorithm was operating at O(n^n^n) or something. What can be done?
Doing It Wrong
I took a look at my assemble strips function. After poking around for a while, I zeroed in on something that had been bugging me for a while. I had been using a function
MagickResetImagePage combined with
MagickCoalesceImages to composite images over each other. I had decided to use these functions before I knew this operation was called “compositing”, and I had found them in a tutorial on making animated .gif files in MagickWand. At the time, I was never really happy with this implementation, so I went back to the API docs to see if there was a function with “composite” in its name. There was.
MagickCompositeImage is a lot more intuitive to use than
MagickResetImagePage. It doesn’t have that Magickal formatting string that
MagickResetImagePage uses, it just takes coordinates. Perhaps this was the solution to my problem. I refactored, and recompiled.
Measure, Don’t Guess
That old gem: I’m sure you’ve heard it too. I decided that maybe this was my best course of action. I decided it was time to learn how to use this Valgrind thing all the Cool Kids are talking about these days. For those of you not in the know, Valgrind is a utility that will tell you various things about your program. The most important/most well-known thing that it can do for you is identify memory leaks. Thinking that prehaps I have a memory leak, I installed Valgrind and got to work.
It turns out that GTK has more than a few memory leaks. Allegedly this is due to the fact that it doesn’t cleanup on exit, relying on the OS to free the memory on program termination. While the general consensus is that this is fine, it doesn’t help us. The folks at Gnome are aware of this, and there is even a Wiki page on ways to mitigate this. The cliff’s notes version of that page being: “Just search for ‘definitely lost'”.
Armed with this piece of wisdom, I set off. I ran the Photo Booth in Valgrind, and examined the results. Valgrind actually turned up some memory leaks, which I corrected. Maybe now we’re set!
Breaking Out The Profiler
This is what they usually want you to do when they tell you to Measure. Unfortunately for me, NetBeans’ built-in profiler is only for Java. After some google searching, I found gprof. Gprof is a pretty bare-bones profiler. It does what it says and not much else, which is fine. I hooked my program into the profiler and got to work. The results? Nothing. My two GTK idle functions ran some 7 million times, returning basically immediately each time as expected. Every other function performed as expected.
Trying The Process Monitor
Having run through Valgrind and GProf, coming out empty-handed, I was at a loss. I got into development because I wanted to fix my own broken code instead of mitigate somebody else’s, and fix it I will. Luckily I have 10 years of sysadmin experience to fall back on. I dusted off my process monitor and got to work.
I fired up DMP Photo Booth, and watched it in the process monitor. I pushed the button. I pushed it again. And again. memory use rose and fell predictably as the strip was assembled, but CPU usage stayed relatively low. Then boom!
I tried again, this time doing literally nothing. Still my computer sputtered and died. I killed the process, but again it was too late.
But wait, isn’t the OS supposed to clean up after me when my process ends? Something fishy is going on.
Have I Mentioned That Threads Are Hard?
Having eliminated all other possibilities, I was forced to consider that I was having a threading issue. “But I was so careful!” I thought. Shortly thereafter I noticed it: I was getting random pthread mutex errors on my console. Clearly I had a threading issue on my hand. Was I spawning extra threads? Was something not releasing its lock? Was I being victimized by gremlins? I set a break point on line one of main() and fired up my debugger. It was time to see just what was being done when nothing was being done.
So, I stepped through my program. Whenever I got to a
g_thread_new call, I made sure the thread function was solid. Finally, I got to my
g_idle_add calls. I had two of them, one to monitor the status indicators, and one to retrieve photo strip thumbnails. Both of these functions pop from a result from a GAsyncQueue. These Queues are fed by worker threads. I thought back to my profiler output and remembered how often these are called. Looking a few lines down I saw a call to
g_timeout_add_seconds. This function is basically adds an idle function, but is only called at most X seconds. Maybe replacing the
g_idle_add calls with
g_timeout_add_seconds was my answer. I refactored and reran.
Well, crud. “Are these functions even my problem?” I thought. I commented them out, recompiled and reran.
“So, what’s the difference?” I wondered. All three of these functions rely on the same basic behavior: pop from a GAsyncQueue some result placed there by a worker thread. I looked at the three threads: the thread that was working properly calls
g_async_queue_ref/unref, and the two that don’t work do not take a reference, instead accessing the static global variable in their module. I refactored all thread functions that access a GAsyncQueue to take a reference and work on their local copy only. I recompiled, reran, and went to bed. 46,100 seconds later, everything was humming along just fine.
Wait, So I Just Had To Increment A Reference Count?
It certainly seemed odd. That’s like your car not starting if the headlights are out. Sure, they’re important, but the car should still start right?
Looking through the source of glib didn’t help. So far as I can tell, all that does is increment the reference count, and return a pointer. I turned to the documentation, which says “… Whenever another thread is creating a new reference of (that is, pointer to) the queue, it has to increase the reference count (using g_async_queue_ref()). Also, before removing this reference, the reference count has to be decreased (using g_async_queue_unref()). …” While not definitive, this certainly seems to indicate that taking a reference is important.
Frankly, I’m not happy about this answer. This is just the sort of magic solution that I hate; it’s fixed, but I’m not sure why. For the time being, I won’t dwell on it. Moving forward, I’ll be sure that my threads take a reference of a GAsyncQueue before calling methods on it. At some point when all of this is said and done, perhaps I’ll investigate this mysterious reference count.
I have taken away from this a new appreciation of just how brittle threads are. Sure, they are powerful, but shooting yourself in the foot with a 50 cal hurts a lot more than with a 9 mm. I’ll have to be more careful.
It was also a good introduction to GProf and Valgrind. Expect blog posts on the usage of each of these tools soon!
Well, it’s been a long time coming, but this weekend 45 days of work has culminated in my first milestone: DMP Photo Booth has received its first commit. If you go to this page, you’ll find the newly created Github repository for it. Feel free to go judge all the choices I made, I’ll wait…
…done? Good, now I’d like to use this post to talk about it.
The Story So Far
So far, most of the work is done in the core. There are module stubs for each module, and they are loadable and “function”, but they don’t do much of anything. However, the UI for the core is, for the most part, done. If you build the core (Either open the projects in NetBeans or cd into its directory and enter
make build), and run it (as of this writing, it must be run from the project root directory, since the supporting files are located here and referenced by relative paths) it will launch. At this point, you can launch the about window, the config window, or exit the program. From the config window you can load your modules (built stub modules are included for your convenience), and the status indicators on the main window will detect it.
All this is fine and good, but the real story here is the plumbing that has been laid.
Not much works at the moment, but the groundwork has been laid for things to come. Going down the list:
In this header, you’ll find all the magic integer constant return values used throughout DMP Photo Booth and friends. I touched on this topic briefly in a previous post, but the cliff’s notes version is that I feel that using a bare integer instead of an enum reduces the burden on module developers, and that the benefits of an enum aren’t great enough to justify this burden.
As this header has no accompanying .c file, it can be
#included by module developers so they can make use of it if they like. I do this in my modules.
One of the first things I worked on was module handling. This header holds all the functions required for the loading and unloading of modules, and the calling of module functions. At first I was implementing all of this with POSIX calls, but it occurred to me that I was defeating the purpose of my modular architecture by using non-portable calls. For this reason, I switched to GModule for module handling. I wrote a post about this if you want to hear more.
This header contains functions that will be added to modules as callbacks. Currently, this is the function that will be called when the user presses “the button”, and the function that will be called to push a message to the console queue.
My general approach to function design is that functions should not return void. All functions that would, instead return gint. This way, I have a mechanism to detect error conditions. In a language without Exception Handling, this is a nice thing to have. This header contains functions to extend this. Since sometimes you can’t return gint, I’ve created a facility to log error codes, similar to how the POSIX errno function works.
This header contains a thread safe queue to store messages to print to the console. Currently, this is implemented with a GQueue and mutex locking, but I made that choice before I knew about GAsyncQueue. I may refactor this at some point to use GAsyncQueue, but for now it works fine, and I see no immediate issue with my implementation.
The idea behind the console queue is that there should be a generic way to store messages. Currently the console is a GtkTextBuffer, but if I build around this, it limits future functionality. Suppose I wanted to support a command line only option? It’d be silly to require GTK for that, wouldn’t it?
That said, the default implementation uses a function placed in the GtkMainLoop to pop all messages out of the queue and place them into the GtkTextBuffer.
This header file is in charge of keeping those fancy module status indicators up to date. There are two main moving parts here: a thread that polls the module status, and a function run by the GtkMainLoop.
The Thread polls the modules every 3 seconds and pushes any status changes to a GAsyncQueue. Meanwhile the GtkMainLoop checks the GAsyncQueue for updates, and sets the indicator status accordingly. Why doesn’t the GtkMainLoop just check the module status you ask? I wanted to make sure the GtkMainLoop does as little work as possible. Currently the calls to check module status are pretty trivial, but there is no guarantee that they will stay that way. Since nobody likes an unresponsive interface, I kept them separate.
This rather hefty header contains two things: a function to build and launch the UI, and all the callbacks called by said UI. All the callbacks are named according to their glade widget name to maintain some semblance of readability.
The Next Step
Obviously, looking through the User Interface source, there are signals to be hooked up. But aside from that, the core is coming along. The one big portion of the core remaining to be done is the photo booth strip logic. The camera module will take X pictures and download them. At this point it is up to the core to assemble them into on image file to be printed.
For now, I will continue to hook up signals and refine the existing portions of the core. While I do this, I will research image handling libraries. I’d like to keep things within the GLib family, but failing this, I will try to keep it limited to libraries shipped with Ubuntu (The default platform). Windows compatibility will come, but version 1.0 will work on Ubuntu.
Also looming on the horizon are the modules. I have a general idea of how I will deal with these, but nothing is set in stone. The first module that I will likely handle is the trigger module. I plan to implement the trigger module using Arduino. This is a topic that interest me, but I’ve never had a good project idea. Whenever I get tired of dealing with GTK, it should be a nice change of pace.