Bramz’ warehouse

Yesterday, I had some fun playing around with remote debugging and examining crash dumps. In the lab, we’re currently running a lot of noise map simulations for a project we’re involved in. These simulations are distributed over all available computers in the lab, a mixture of old and new, often actively used by other people as desktop. The fun part starts when things do not run as smoothly as planned, which is unfortunately often the case. What else to expect expected from software that is simultaneously developed and used in production. We’re a research lab, after all.

Our current noise mapper is pretty young and only supports full multi-threading since recently. Since we use this to squeeze out that extra bit of performance on machines that have more than one core, we started noticing - not surprisingly - inconsistent and hard-to-reproduce crashes. Reproducing the crashes on the development box was not feasible as you could just spend hours waiting for the crash to happen. Debugging the crashed programs locally wasn’t an option either, as most of them didn’t have any debugging tools installed whatsoever.

Last week, I’ve found some time to investigate the problem further, and I first started poking around with remote debugging in Visual Studio 2005, but it wasn’t really helpful. Normal use requires that you have identical accounts on both PCs. In the heterogeneous environment of our lab, without central user management and where everyone admins his own desktop PC, this is not a easy thing. However, it also has a no-authorization mode (who needs security anyway =), and after disabling the firewall (the rule automatically created by the remote debugger somehow wasn’t enough) and modifying some obscure security setting, I finally was able to connect to it from my dev box. Unfortunately, I only got corrupted call stacks, so it wasn’t of any help at all.

But that’s when I noticed something really wonderful: minidumps. When your program crashes the hard way on WinXP and later, it pops up a dialog box asking if you want to send debug information to Microsoft. You of course don’t want to do that, but at that moment, a minidump already has been created. You just need to secure it before you click away the dialog box. Then you copy it to your dev box, open it in visual studio and press F5. Et voila, you get an instant reproduction of the crash, ready to be debugged. Then it was almost a breeze to solve the problem.

Why have I never noticed this before? It’s so helpful to investigate problems that happen on machines other than yours. It’s plain magic. I’m thinking of building it in in our software so that the crash dumps are automatically send to me. If I also can avoid the dialog box, I can run the simulations unattended, and still be reported of any issue. When a crash happens, the program simply sends me the crash dump and tries to run the next job. I already have something similar for problems that arise in the Python code of the simulations, but this would make it complete.

Perhaps to be continued =)

I’ve bought myself a Canon EOS 400D recently. It’s a digital single-lens reflex camera what means it uses a mirror to show the image in a optical viewfinder instead of showing them on an LCD, and that you can also replace the lens. I’m pretty happy with it, it’s a nice machine that gives you total control of the image you want to shoot. I only have the standard kit lens on it now, which is of somewhat minor quality. But good lenses are expensive, so I’ll learn to take better pictures first =)

Anyway, I decided to set up a photogallery because taking pictures without showing anyone is a bit … well … senseless. I’ve also made an LiAR album where I’ll upload some high definition renders made using my own renderer LiAR isn’t a raytracer. Unfortunately, I seem to have lost many of them =(

If you haven’t noticed yet, the new Gaim, now know as Pidgin 2.0 has arrived after months of beta testing (the link seems to be currently slashdotted =). And it rocks! As you might know, the reason why it took so long was due to a legal dispute with AOL over the name Gaim (that sounds like AIM), preventing the developers to release new versions or make significant changes. It also resulted in the name change.

For people who’re not familiar with Pidgin or Gaim, it’s a multiprotocol instant messenger application. So if you’re find yourself in the position of using MSN, GoogleTalk, ICQ, IRC, yabber or other protocols at the same time, you might want to give Pidgin a try! (direct download link in case their homepage is still slashdotted)

pycpuid is a very simple Python extension. It reads the information available from the CPUID assembly instruction, and makes it available to any Python program. I needed it to decide on some codepath based on whether the box supported SSE2. I didn’t found anything alike, so I coded it myself.

Here’s a quick roundup on how to build Python extensions using the Microsoft Visual C++ 2005 Express Edition compiler. This is meant for people who need to build Python extension from source using the accompanying setup.py script.

There’s no rocket science or magic involved. Just five easy click-through steps. Well, the last one actually is a bit of manual editing, but not so much. And the best thing of all: it’s all available for free!

1. Probably you will already have Python installed, but for the sake of completeness: download and install Python (get one of the Windows Installers)
2. Download and install VC++ 2005 Express (pretty obvious)
3. At the end of the VC++ 2005 Express installation, you will be asked to register your copy. Please, do so. It’s for free, and if you don’t do it, your copy will stop working after a week or so. If you’ve skipped registration, you can still do it from the Help menu by clicking on Register Product.
4. VC++ 2005 Express is a limited edition and is not capable of building Win32 applications out of the box (it can build .NET applications though). For the most simple extensions you probably won’t need that, but sooner or later you will hit that limitation. Fortunately, you can solve that issue (for free again) by installing the Windows Platform SDK. If you follow the previous link, you will notice it actually goes a download page for the Windows Server 2003 R2 Platform SDK. That’s OK, it’s exactly what you need. Caution: if you go looking for the Windows Platform SDK via google, you probably will find a previous version. So just use that link I gave you.
5. By now, you have installed on your computer everything you need to build Python extensions using Visual C++ 2005 Express Edition. All that’s is left to do is to update some settings in VC++ 2005 Express so it can find the necessary headers and libraries. Open up the Options dialog from the Tools menu. On the left, select VC++ Directories from the node Projects and Solutions. In the top-right corner, you’ll then find a dropdown box titled Show directories for. Add the following directories:
   • Executable files:
     • C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2\Bin
     • C:\Python25
   • Include files:
     • C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2\Include
     • C:\Python25\include
   • Library files:
     • C:\Program Files\Microsoft Platform SDK for Windows Server 2003 R2\Lib
     • C:\Python25\libs

That’s it! Enter setup.py install and go!

Wim Verbrugghe, and old-time pal and conceptartist/animator, started a blog womstadt on which he will hopefully show us some nice art =) He’s currently involved in a team that will help shape the Top Secret project from acclaim.

update: He also has a deviantart page now. I’m curious which one will survive =)

Yesterday, while I was visiting the VirtualDub website, I bumped into an interesting post about Frame Pointer Omission (FPO) optimization and why it renders perfect stack walking impossible when combined with callee-pops parameter passing.

Recalling my issues with stack walking in a profiling tool of Max McGuire (I couldn’t get credible call graphs out of it), I checked my settings and found that, indeed, FPO was enabled. I disabled it, and hey, suddenly it worked!

It’s a bit weird though, because I’m using __cdecl which is caller-pops, plus I had full PDB information. So it should have worked even with FPO enabled.

Anyway, if you want to profile your app with call graph information, and you don’t get credible results, try disabling FPO.

Perhaps to be continued …

The last couple of weeks I’ve mostly been coding on pyshapelib, that’s a small Python wrapper around shapelib, a C++ library to read ESRI shapefiles. It is originally written by Bernhard Herzog and is now maintained as part of Thuban, an open source interactive geographic data viewer. My contribution to pyshapelib is rewriting it to use hand-crafted Python bindings instead of SWIG generated ones, to support shapefiles with height (Z) or measurement (M) information, and to have better support for Unicode strings.

Until now it’s been quite interesting. I’ve learnt some new things about writing Python extension modules, for example that METH_VARARGS functions do not take keyword arguments (something we assumed otherwise in the Lass Python API =) (update: it seems we’ve gotten that right after all). But the best surprise was this one:

mixing msvcr71.dll and msvcr80.dll can be really inauspicious

I’m not yet sure what exactly is going on, but when I tried to import my freshly built wxproj (part of Thuban) in Python, LoadLibraryEx in dynload_win.c failed resulting in a nice error message in the style of …

    from wxproj import point_in_polygon_shape, shape_centroid
ImportError: DLL load failed: A dynamic link library (DLL)
initialization routine failed.

… and a nice message box:

It’s always nice to get the advice to contact yourself for more information, it’s so helpful =)

Opening wxproj.pyd in Dependency Walker revealed that wxproj.pyd was dependent on both msvcr71.dll (through wxmsw26uh_vc.dll of wxPython 2.6) and msvcr80.dll (through proj.dll of PROJ.4). wxmsw26uh_vc.dll came straight out of an installer, but proj.dll I have compiled myself.

From there, it was easy guess work. I replaced proj.dll with a version compiled with VC7.1, and the problems mysteriously vanished.

Thing learnt: when building Thuban yourself, strictly use DLLs built with VC7.1 =)

It still makes me wonder though. I’ve compiled extension modules with VC8 before, and I’ve never encountered this problem before. And yet, they all must have been indirectly dependent on msvcr71.dll through python24.dll (Python 2.4 is compiled with VC7.1). So why is it complaining in wxproj’s case only?

Perhaps to be continued …

update: (04/04/2007) The error was caused by a missing manifest in proj.dll. The issue is finally resolved by changing the makefile.vc to embed the manifest, as following:

proj_i.lib:     $(LIBOBJ)
    link /debug /dll /def:proj.def /out:$(PROJ_DLL) \\
        /implib:proj_i.lib $(LIBOBJ)
    if exist $(PROJ_DLL).manifest mt.exe -manifest \\
        $(PROJ_DLL).manifest -outputresource:$(PROJ_DLL);2

A ticket has been submitted to PROJ.4’s BugZilla.

Using an image licensed under a share-alike creative commons license implies that I must make my wordpress theme available for others as well. And so I did. You can find my Warehouse Theme as a seperate download in the miscellaneous section.

I’ve updated my article “Reflections & Refractions in Raytracing” with some typo fixes. In the source code of rSchlick2, there was a cosI where it must be a cosX. Kudos to Anders Bennett.