Or a router along the way. Internet traffic takes a lot of "hops" and its pretty difficult to guarantee "quality of service" along the entire path.
If most other users are reporting no problems it is unlikely that the problem is SMF.
At a DOS command prompt, type
This will give you an idea of how many routers the traffic is crossing to get there and back. My ISP is 18 hops away. A single device along that route malfunctioning can slow things way down.