For me, its a taste test. but, if they get to the point that they don't look real healthy and start to shrival a little, they are getting ready to fall off. At that point, it's a little late.
As for the taste test comment, you get a feel as you get used to growing them. A young, imature pepper may look like it's ready but it won't have any heat or very little. A mature pepper may look look entirely different but carry a lot of heat. You just have to understand the type of pepper plant you're growing, climate, temps, etc.
Lastly, I grow Thai Dragons every year. Personally, I don't find any taste difference from a mature green Thai and a more mature red Thai. So, I pick them while they're green and mature and then I get bigger yields as they regenerate quicker.