ThispaperhasbeensubmittedforpublicationonNovember15,2016.
LearningfromSimulatedandUnsupervisedImagesthroughAdversarial
Training
AshishShrivastava,TomasPster,OncelTuzel,JoshSusskind,WendaWang,RussWebb
AppleInc.
{a_shrivastava,tpf,otuzel,jsusskind,wenda_wang,rwebb}@apple
Abstract
Withrecentprogressingraphics,ithasbecomemore
tractabletotrainmodelsonsyntheticimages,poten-
tiallyavoidingtheneedforexpensiveannotations.How-
ever,learningfromsyntheticimagesmaynotachievethe
desiredperformanceduetoagapbetweensyntheticand
realimagedistributions.Toreducethisgap,wepro-
poseSimulated+Unsupervised(S+U)learning,where
thetaskistolearnamodeltoimprovetherealismof
asimulatorˉsoutputusingunlabeledrealdata,while
preservingtheannotationinformationfromthesimula-
tor.WedevelopamethodforS+Ulearningthatuses
anadversarialnetworksimilartoGenerativeAdversar-
ialNetworks(GANs),butwithsyntheticimagesasin-
putsinsteadofrandomvectors.Wemakeseveralkey
modicationstothestandardGANalgorithmtopre-
serveannotations,avoidartifactsandstabilizetraining:
(i)aself-regularizationˉterm,(ii)alocaladversarial
loss,and(iii)updatingthediscriminatorusingahistory
ofrenedimages.Weshowthatthisenablesgenera-
tionofhighlyrealisticimages,whichwedemonstrate
bothqualitativelyandwithauserstudy.Wequantita-
tivelyevaluatethegeneratedimagesbytrainingmod-
elsforgazeestimationandhandposeestimation.We
showasignicantimprovementoverusingsyntheticim-
ages,andachievestate-of-the-artresultsontheMPI-
IGazedatasetwithoutanylabeledrealdata.
1.Introduction
Largelabeledtrainingdatasetsarebecomingincreas-
inglyimportantwiththerecentriseinhighcapacitydeep
neuralnetworks[4,18,44,44,1,15].However,labeling
suchlargedatasetsisexpensiveandtime-consuming.
Thustheideaoftrainingonsyntheticinsteadofrealim-
ageshasbecomeappealingbecausetheannotationsare
automaticallyavailable.Humanposeestimationwith
Kinect[32]and,morerecently,aplethoraofothertasks
havebeentackledusingsyntheticdata[40,39,26,31].
Rener
Unlabeled Real Images
SyntheticRened
Figure1.Simulated+Unsupervised(S+U)learning.Thetaskis
tolearnamodelthatimprovestherealismofsyntheticimages
fromasimulatorusingunlabeledrealdata,whilepreserving
theannotationinformation.
However,learningfromsyntheticimagescanbeprob-
lematicduetoagapbetweensyntheticandrealim-
agedistributions¨Csyntheticdataisoftennotrealistic
enough,leadingthenetworktolearndetailsonlypresent
insyntheticimagesandfailtogeneralizewellonreal
images.Onesolutiontoclosingthisgapistoimprove
thesimulator.However,increasingtherealismisoften
computationallyexpensive,therendererdesigntakesa
lotofhardwork,andeventoprenderersmaystillfailto
modelallthecharacteristicsofrealimages.Thislack
ofrealismmaycausemodelstooverttounrealisticˉ
detailsinthesyntheticimages.
Inthispaper,weproposeSimulated+Unsupervised
(S+U)learning,wherethegoalistoimprovethereal-
ismofsyntheticimagesfromasimulatorusingunla-
beledrealdata.Theimprovedrealismenablesthetrain-
。。。以上简介无排版格式，详细内容请下载查看