By default, when you enable the Group similar taskbar buttons option in Taskbar properties, items are grouped only when the taskbar buttons begin to get too small, and then the item that you opened first, is grouped first.
To change this behavior, create a "TaskbarGroupSize" DWORD entry in the following registry key:
HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersionExplorerAdvanced
The values for this key are:
0 – (Default) Group by age (oldest group first)
1 – Group by size (largest group first)
2 – Group any group of size 2 or more
3 – Group any group of size 3 or more
Log off and then back on for this change to take effect.
Reference: http://support.microsoft.com/kb/281628/en-us