We address the problem of image feature learning for the applications where multiple factors exist in the image gen- eration process and only some factors are of our interest. We present a novel multi-task adversarial network based on an encoder-discriminator-generator architecture. The en- coder extracts a disentangled feature representation for the factors of interest. The discriminators classify each of the factors as individual tasks. The encoder and the discrimina- tors are trained cooperatively on factors of interest, but in an adversarial way on factors of distraction. The generator provides further regularization on the learned feature by re- constructing images with shared factors as the input image. We design a new optimization scheme to stabilize the ad- versarial optimization process when multiple distributions need to be aligned. The experiments on face recognition and font recognition tasks show that our method outper- forms the state-of-the-art methods in terms of both recog- nizing the factors of interest and generalization to images with unseen variations.