Try leaving some of the old charred wood chunks in the smoke box, then, put your fresh wood on top. It may take a while longer until the first smoke starts, but it slows things down quite a bit. I've made this a habit in the last 6 months or so. It seems to work pretty reliably with my small GOSM. I also have switched to using mostly small/medium sized chunks instead of chips.
My rig has no lower vents, just the burner housing intake, which is non-adjustable, and the upper vent, which I do at times need to pinch down to a near closed position to cut the draft way back in extreme circumstances (cold/windy). I try to run it just far enough open to keep the burner running quite. If I open too far, the burner will start to rumble from too much draft. The higher burner flame will definately effect the amount of heat absorbed into the smoke box. That's just the nature of the beast.
You have to compensate for the additional BTU output needed for wind/cold somehow, and there are several ways...some are mentioned in other replys...this is just how I do it...I try whatever comes to mind at the time, and stay with it if it works.
The tabs/stoppers on the vent control can be bent out and flattened so that the vent will close farther...learned that trick the first month I used mine.
Oh, and don't forget to block the wind as best you can...wind kills outdoor cooking appliances, they have to run with so much hotter fires to maintain temps.
Hope that helps...good luck, and happy smokin'!!!