Yes you do need to watch them to ensure that they don't burn. You want to avoid direct heat on the nuts for too long or they can burn. I hot smoke mine in an FEC-100 which provides heat from the top and I have a large pan on the top shelf to act as a heat baffle and each time I stir the nuts I rotate their position in the smoker. This avoids hot spots. You are looking for the faintest of straw colour on the cashews and no real colour on the almonds. When I demonstrate this in my BBQ courses I use a ProQ vertical smoker or a WSM with an empty water pan in place and these do not burn.