So many people get caught up in the “I must deal with the factory only!” mentality. There's some things to consider if you want to deal only with a factory.
1) If you're dealing with the factory, they're going to expect a lot larger orders. Are you prepared to make a considerable order?
2) The level of western-friendliness is generally much lower with factories opposed to trading companies. Horrible or non-existent English is to be expected. Are you going to feel comfortable facing such communication barriers to save 5% or so? Which brings us to the second point…
3) The amount of money you save going through the factory is generally quite small. You might only save between 5-10% or worse, you might save none because the trading company is able to negotiate such low discounts that they can give you an even lower price than if you went directly through the factory.
4) Almost all Chinese factories outsource at some point parts of the manufacturing process. So the issue of who and who isn't a factory is blurred.
If you're importing such a vast quantity of product that 10% means something to the order (10% of $5000 is still a pretty meager amount for the potential inconvenience of dealing with a factory; 10% of $500,000 on the other hand is a different story) dealing with a factory might be the way to go. Nevertheless, I did a post a while back on distinguishing between trading companies and factories here: Trading Company or Factory?.