Category Archives: Software

Double page numbers in XHTML2PDF/Pisa

I’ve been working on a django project recently that generates PDFs from html using the XHTML2PDF library (formerly Pisa). I have html successfully in a pdf in just a couple lines, so overall it’s fantastic, but there are some quirks to get around. I ran into one of these quirks today while trying to put page numbers in my footer template – when it rendered, every page was showing its page number twice (11, 22, 33). My code looked fine; no extra tags, everything was properly formatted, etc, but I could not stop it from happening. Here is my page/footer style and my extremely simple footer markup.

@page {
	size: {% if pagesize %}{{pagesize}}{% else %}letter{% endif %};
	margin: 1cm;
	@frame footer {
		-pdf-frame-content: footerContent;
		bottom: 1cm;
		margin-left: 1cm;
		margin-right: 1cm;
		margin-top: 0cm;
		height: 1.4cm;
	}
}

...

<div id="footerContent">Page <pdf:pagenumber /></div>

Finally I got smart and found this answer on the mailing list. Essentially its a bug in the xhtml2pdf parser – without a new line after the tag it freaks out and decides you meant it twice. Interesting bug, but not interesting enough for me to go look at their parser code to figure it out. :)

Anyway, this simple change fixed the problem and hopefully you won’t waste any time banging your head on the desk with page 11, 22, etc.

<div id='footerContent'>
 Page <pdf:pagenumber />
</div>

MochaUI Custom Tiling

Our flagship product over at the dayjob is a webapp built on Mootools and MochaUI. Mootools makes everything work and Mocha gives us a dashboard with widgets and windows and all that fancy stuff.

To start with I was simply using the Tiling script that comes with mocha out of the box. It grabs all the windows, counts them up, and distributes them evenly across the screen (you can see it at their demo – open a bunch of windows and hit View->Tile Windows). This worked wonderfully for me (23″ monitors) and the VP (30″ monitor), but some of the testers with smaller computer screens (13″ craptop) were complaining that their windows weren’t ‘good’. When the dashboard was that small, opening more than a couple widgets and tiling them resulted in things overlapping or offscreen and all the windows too small to be useful.

The suggestions started rolling in about having a fixed layout instead of a dynamic one based on the number of windows you have open, but nobody agreed on the ‘proper’ numbers. Some with really small screens suggested 2 wide and only 1 tall. The majority were in the 2×2 – 2×3 range. Mr. 30″ screen was comfortable with windows up to 3×3. Clearly we needed a better solution.

Introducing the tile layout selector. New accounts get started with 2×1 and a simple walkthrough so even the tiniest screens should survive the instructions. After that, everyone has this on their toolbar.

Tile Layout Selector and Auto-Tile Option

Now you can choose which layout works best for you. I’m sure 30″ monitor VP has his set on 3×3, most are on 2×3, and the craptop guy has 2×1. Everybody is happy, everybody can read their widgets, and all the lay people think it’s a really clever feature. Additionally, you can see the ‘Auto-Tile Widgets’ checkbox in the image. That’s another feature you should consider adding. Personally, I detest it re-tiling my windows all the time, but the non-technical people seem to love it. Keep it as an option and nobody should be upset.

Now implementing this isn’t terribly hard, but it isn’t trivial either. One of the new problems you have to solve is where to put additional windows that go beyond the user selected layout. For example, if the user picks 2×2, that creates 4 ‘slots’ on the dashboard. Where does the fifth tile go? The default Mocha script makes the right decisions for rows and columns based on the total number of tiles but, since we are giving it the layout, it will just start running additional windows off screen.

Step one in my solution is to break the dashboard into the ‘slots’ mentioned above and simply loop through them placing windows. In the above example, the fifth window in the 2×2 layout would loop back around and get placed in the top left slot again. This keeps things from being placed off screen, but just overlapping the other windows isn’t very useful.

Step 2 combines step 1 and cascading. When the windows loop back around, I want them to cascade inside their ‘slots’ so we can still see all the title bars and everything stays nice and tidy. First we calculate the maximum number of overlaps we will have in any ‘slot’. Then we take that number and use it to scale some additional padding we are going to use when calculating the new tiled window size. This will keep a long stack of cascaded windows from overflowing their tile slot and looking bad. Then we place the tiles as normal but every time we loop back to the top we increase the top and left offsets a bit so they automatically get cascaded.

The result is a nice looking fixed-layout tiling that can handle a large number of additional windows beyond the specified layout without losing window visibility or aesthetic value.



Here is the code so you can implement this yourself. The placement and sizing code is almost the same as the default mocha script and the rest is fairly well commented so you should be able to see what is going on. You may notice I also added firing resize events to each window when it gets tiled. We save the user’s window locations and sizes every time they get moved, so this lets the system save their windows when they get tiled as well. Depending on how your dashboard looks, you’ll probably want to tweak the padding numbers – especially loopoffsetx and loopoffsety. These are what get added every time the script loops to make the windows ‘cascade’ down from the previous window. Play with them until you’re happy.

Note: You’ll need some way to get the layout you want — the code below looks for a dom element with ID ‘dashboard-tilelayout’ and gets its value (the dropdown pictured above). Change to suit your own needs.

This is my entire arrangeTile function (with some non-relevant stuff stripped out). It comes from the original function by Greg Houston and Harry Roberts (thanks guys!) which is included in the Mocha distribution.

arrangeTile: function(){
 
// some offsets for dashboard and dock items
var viewportTopOffset = 180;
var viewportLeftOffset = 40;
var x = 10;
var y = 130;
 
// get the desired layout from the tilelayout dropdown on the dashboard
var tilelayout = $('dashboard-tilelayout').value;
var cols = parseInt( tilelayout.substring(0,1));
var rows = parseInt( tilelayout.substring(2,3));
 
// get all the window instances
var instances =  MUI.Windows.instances;
 
// count the windows that are not minimized, not maximized, and additionally have isWidget = true
var windowsNum = 0;
instances.each(function(instance){
	if (!instance.isMinimized && !instance.isMaximized){
		windowsNum++;
	}
});
 
// calculate how much extra to space widgets based on total slots vs total widgets visible
var slots = cols * rows;
var overlaps = Math.floor( windowsNum / slots);
 
// increment the 'slot' padding based on overlaps
var widthpadding = 5;
var heightpadding = 5;
for(var i=0; i<overlaps; i++)
{
	widthpadding += 20;
	heightpadding += 20;
}
 
// figure out where the tile slots go
var coordinates = document.getCoordinates();
var col_width = ((coordinates.width - viewportLeftOffset) / cols);
var col_height = ((coordinates.height - viewportTopOffset) / rows);
 
var row = 0;
var col = 0;
 
// play with these until you're happy
var loopoffsetx = 15; // how far to offset the next widget that hits the same tile 'slot'
var loopoffsety = 25; // vertical offset
var offsetx = 0; // start at zero -- this gets increased every loop to make the cascade look
var offsety = 0; // start at zero -- this gets increased every loop to make the cascade look
 
instances.each(function(instance){
	if (!instance.isMinimized && !instance.isMaximized && instance.options.draggable){
 
		var content = instance.contentWrapperEl;
		var content_coords = content.getCoordinates();
		var window_coords = instance.windowEl.getCoordinates();
 
		// Calculate the amount of padding around the content window
		var padding_top = content_coords.top - window_coords.top;
		var padding_bottom = window_coords.height - content_coords.height - padding_top;
		var padding_left = content_coords.left - window_coords.left;
		var padding_right = window_coords.width - content_coords.width - padding_left;
 
		// This resizes the windows
		if (instance.options.shape != 'gauge' && instance.options.resizable == true){
			var width = (col_width - widthpadding - padding_left - padding_right);
			var height = (col_height - heightpadding - padding_top - padding_bottom);
 
			if (width > instance.options.resizeLimit.x[0] && width < instance.options.resizeLimit.x[1]){
				content.setStyle('width', width);
			}
			if (height > instance.options.resizeLimit.y[0] && height < instance.options.resizeLimit.y[1]){
				content.setStyle('height', height);
			}
 
		}
 
		// place the window, including the offsets for cascading
		var left = (x + (col * col_width)) + offsetx;
		var top = (y + (row * col_height)) + offsety;
 
		instance.drawWindow();
 
		MUI.focusWindow(instance.windowEl);
 
		if (MUI.options.advancedEffects == false){
			instance.windowEl.setStyles({
				'top': top,
				'left': left
			});
			// fire resized event
			instance.fireEvent('resize', instance.windowEl);
		}
		else {
			var tileMorph = new Fx.Morph(instance.windowEl, {
				'duration': 550,
				'onComplete': function(){
					// fire resized event
					instance.fireEvent('resize', instance.windowEl);
				}
			});
			tileMorph.start({
				'top': top,
				'left': left
			});
		}				
 
		// if at the right side of the screen
		if (++col === cols) {
			// reset and move down one row
			col = 0;
 
			// if at the bottom of the screen
			if(++row === rows)
			{
				// reset to top and increase the offsets
				row = 0;
				offsetx += loopoffsetx;
				offsety += loopoffsety;
			} // end if(++row === rows)
		} // end if (++col === cols)
 
	} // end if this is a window we want to tile -- !minimized, !maximized,  isDraggable
}.bind(this));
} // end function arrangeTile

Chronolapse 1.0.4

Chronolapse has hit version 1.0.4!

The last version was built without the console window, but when frozen into an executable this was causing the mencoder process to freeze indefinitely. This addresses the issue, fixes the update checking code, and makes the mencoder feedback not hang the GUI while it runs.

Just in time for LD!

http://code.google.com/p/chronolapse/

Interesting python + MySQL speed test

I’m currently working on a project involving a large amount of database processing on the backend. The last week has been dedicated to getting a script running to take the crappy data from multiple DBs, process it into something nice, and inserting it into my working database. The original stab at the code with no attempt to speed it up wound up taking about 4 seconds per processed item. This particular database had about 170,000 items to process, so the import script would take something near 189 hours. I broke out the profiler and found over 99% of the time was wasted while waiting for mysql select queries to come back. My initial round of changes helped immensely (from 4.0 to 0.25 seconds per item), but we were still at almost 12 hours to run. That’s when I ported the code to python and started testing threads.

Before I get into it, let’s make this crystal clear: THIS IS NOT GREAT TEST DATA. DRAW NO CONCLUSIONS FROM THIS. This was not a robust test in any way. I continued to make time saving changes and bug fixes as they were discovered, so even the time between runs can’t be relied upon. Nor was this on an isolated server — this was done on a production server currently hosting dozens of websites with unknown traffic and load. Even the test numbers are too small to make real assessments of performance. Finally, this applies ONLY to my precise script and the algorithm I’m using. You will get different results with a different project. It would be foolish to take anything concrete from these numbers.

That being said, the results are still interesting to consider and they should encourage you to do your own testing when building your next database heavy project.

Up to this point I had been looking at the time/processed item, so I started there after throwing together the threaded importer. Here are the results of running the script with a set number of entries for each thread to process with varying numbers of threads.

Seconds Per Processed Item
Threads 100 Items per Thread (Total) 250 Items per Thread (Total)
1 .118 (100) .106 (100)
2 .097 (200) .106 (250)
3 .095 (300) .110 (500)
4 .102 (400) .114 (750)
5 .106 (500) .115 (1250)
8 .112 (800)
10 .116 (1000)
12 .120 (1200)
15 .120 (1500)
Seconds/ Item by Items/Thread


Not at all what I expected, but with the vastly different number of total items being processed, I decided to take another approach. Here is the same script, but this time with a set TOTAL number of entries to process (5000). In other words, the above data did X items per thread, so the total number of processed items was threads * X. Now, I’m assigning each thread 5000/threads entries. This makes much more sense for the project (the first test was plain stupid) and, as you can see, the results are more what you would expect.

5000 Items Evenly Across All Threads
Threads Total Time Time per Item
1 08:17.21 0.0994
2 07:13.33 0.0867
3 06:41.97 0.0804
4 06:32.97 0.0786
5 06:27.36 0.0775
8 06:23.59 0.0767
10 06:24.21 0.0768
12 06:24.24 0.0769
15 06:23.25 0.0767
5000 Items - Time/Item by Threads


Now you can see that the threading is making a sizeable difference – almost 23% at its best. An interesting thing to note is that, for this script, the benefits of threading drop off at about 8. Obviously at that point the threads just get in each other’s way instead of keeping the database fully loaded. Overall, the lessons to be learned here are 1. test your project in multiple different ways and 2. threading will likely help out if you’re wasting the majority of your time waiting for database selections (as opposed to processing data).

DokuWiki / WordPress 3.x Integration Fix

As many people noticed, wordpress 3.0 broke the DokuWiki/Wordpress integration I had up. I didn’t find an elegant solution, but I got a hack working and posted it here.