Journal of Textile Research ›› 2023, Vol. 44 ›› Issue (03): 176-186.doi: 10.13475/j.fzxb.20211106611

• Apparel Engineering • Previous Articles     Next Articles

Personalized clothing matching recommendation based on multi-modal fusion

LIU Junping1,2,3, ZHANG Fuhong1, HU Xinrong1,2,3(), PENG Tao1,2,3, LI Li1,2,3, ZHU Qiang1,2,3, ZHANG Junjie1,2,3   

  1. 1. College of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, Hubei 430200, China
    2. Hubei Provincial Engineering Research Center for Intelligent Textile and Fashion, Wuhan Textile University, Wuhan, Hubei 430200, China
    3. Engineering Research Center of Hubei Province for Clothing Information, Wuhan Textile University, Wuhan, Hubei 430200, China
  • Received:2021-11-12 Revised:2022-10-23 Online:2023-03-15 Published:2023-04-14

RichHTML

27

PDF (PC)

170

Abstract:

Objective In the context of fast fashion, most consumers do not have the keen insight of professional designers on fashion clothing matching, which leads to their capability of quickly selecting a set of appropriate, harmonious and suitable clothing from a large number of clothing. In order to better improve users' online shopping experience and help them accurately express their unique personality characteristics, professional identity, status and other image positioning to the outside world, this paper aims to achieve high-precision recommendation by improving the clothing matching degree, so as to meet the huge demand of consumers for personalized clothing matching recommendation.

Method By studying the highly nonlinear complex attribute interaction from clothing color to category, and based on the quantitative standard of matching degree of clothing matching, an embedded model of the potential feature representation space of an item was built. By building a matrix decomposition framework model that integrates multimodal information, the shortcomings of existing multimodal feature fusion algorithms were further analyzed, and the clothing style preferences of different users were depicted. Through feature extraction, multimodal feature fusion match degree was calculated and other operations were carried out to establish personalized clothing matching scheme.

Results PCMF (personalized clothing matching recommendation based on multi-modal fusion) with some conventional clothing matching methods were qualitatively compared. Compared with all baselines, the clothing matching degree calculated by this model reached 0.81, which is 1.25% higher than the AUC (area under curve) value of conventional methods (Tab.2). It is confirmed that the transposition fusion method of text features and visual features used in PCMF improves the correlation between features, making the presentation of individual style more accurate. In order to compare the difference of contribution of different modal information to the matching degree of PCMF modeled clothing, experiments under three different modal combinations were conducted, i.e. PCMF-T (only exploring the text information of items), PCMF-V (only exploring the visual information of items), and PCMF-TV (exploring the visual and text information of items) (Tab.3). The AUC value of PCMF-T reached 0.775, higher than that of PCMF-V (0.763), indicating that the text information of the piece can more succinctly summarize the key features of the piece, such as patterns, materials and brands. PCMF-TV shows better performance than PCMF-T and PCMF-V, which indicates the necessity of combining multi-modal information of items, and verifies the effectiveness of adding user factors to the general clothing matching modeling to make personalized clothing matching recommendation. In order to effectively evaluate the practical application of the PCMF model, the PCMF model was deployed in the complementary item retrieval task (Fig.5). It is demonstrated that the PCMF model can complete the personalized clothing matching recommendation task according to the user's preferences. In addition, the MRR (mean reciprocal rank) measurement method was used as the evaluation index to further evaluate the model. PCMF performs better than other models regardless of the number of clothing candidates (Fig.6).

Conclusion Through the combination of IGCM (item-item general clothing match modeling) and UPCM (user-item personalized clothing match modeling), a personalized clothing matching recommendation model based on multi-modal fusion is constructed, which facilitates high-precision personalized clothing matching recommendation. Specifically, the purpose is to match a lower garment that not only has a good match with a given user's top, but also meets the user's taste. In general, the research results show the necessity and effectiveness of combining visual and text modal information and introducing user factors in personalized clothing matching recommendation and the practical application value of PCMF in real scenes is verified. In the future, the clothing matching recommendation problem of two examples will be transformed into a multi-instance learning problem to provide users with personalized package recommendation including shoes and accessories.

Key words: complementary clothing matching, personalized recommendation, multi-modal, feature extraction, feature fusion, matching degree

CLC Number: 

  • TS941.26

Fig.1

Personalized clothing matching recommendation model based on multi-modal fusion of PCMF structure"

Tab.1

Statistics of data set IQON 3000"

单品种类 数量/件
外套 35 765
上衣 119 895
下衣 77 813
连衣裙 25 816
鞋子 106 598
配饰 306 448

Fig.2

Visual feature extraction process of item"

Fig.3

Text feature extraction process of item"

Tab.2

Performance comparison of AUC by different methods"

方法 AUC值
POP 0. 421
RAND 0. 501
Bi-LSTM 0. 661
BPR-DAE 0. 699
VTBPR 0. 801
GP-BPR 0. 802
PCMF 0. 814

Fig.4

Performance comparison between PCMF and GP-BPR of AUC"

Tab.3

Performance comparison of AUC under different modes"

方法 AUC值
PCMF-V 0. 763
PCMF-T 0. 775
PCMF-TV 0. 814

Fig.5

Bottom recommendation ranking"

Fig.6

MRR performance of six benchmark methods in range T"

[1] AMED I, BALCHANDANI A, BERG A, et al. The state of fashion 2022: global gains mask recovery pains[EB/OL]. [2022-10-01]. https://www.mckinsey.com/industries/retail/our-insights/the-state-of-fashion.
[2] 朱伟明, 卫杨红. 不同情景下服装个性化定制体验价值差异研究[J]. 纺织学报, 2018, 39(10): 115-119.
ZHU Weming, WEI Yanghong. Experiential value differences of clothing personalized customization under different situations[J]. Journal of Textile Research, 2018, 39(10): 115-119.
[3] CHEN Huiyuan, LIN Yusan, WANG Fei, et al. Tops, bottoms, and shoes: building capsule wardrobes via cross-attention tensor network[C]// Fifteenth ACM Conference on Recommender Systems. New York: Association for Computing Machinery, 2021: 453-462.
[4] DONG Xue, SONG Xuemeng, FENG Fuli, et al. Personalized capsule wardrobe creation with garment and user modeling[C]// In Proceedings of the 27th ACM International Conference on Multimedia (MM'19). New York:Association for Computing Machinery, 2019: 302-310.
[5] CHEN Wen, HUANG Pipei, XU Jiaming, et al. POG: Personalized outfit generation for fashion recommendation at alibaba ifashion[C]// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD'19). New York, USA: ACM Press, 2019: 2662-2670.
[6] LIN Yusan, MOOSAEI M, YANG Hao. Outfitnet: fashion outfit recommendation with attention-based multiple instance learning[C]// Proceedings of The Web Conference 2020. New York, USA: ACM Press, 2015: 77-87.
[7] VASILEVA M I, PLUMMER B A, DUSAD K, et al. Learning type-aware embeddings for fashion compatibility[C]// Proceedings of the European Conference on Computer Vision. Berli, Germany: Springer, 2018: 390-405.
[8] CHEN Xu, ZHANG Yongfeng, XU Hongteng, et al. Visually explainable recommendation[J]. Computer Science, 2018. DOI: 10.48550/arXiv.1801.10288.
doi: 10.48550/arXiv.1801.10288
[9] HIDAYATI S C, HSU C C, CHANG Y T, et al. What dress fits me best?: fashion recommendation on the clothing style for personal body shape[C]// Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: ACM Press, 2018: 438-446.
[10] 杨争妍, 薛文良, 张传雄, 等. 基于生成式对抗网络的用户下衣搭配推荐[J]. 纺织学报, 2021, 42(7): 164-168.
YANG Zhengyan, XUE Wenliang, ZHANG Chuanxiong, et al. Recommendations for user's bottoms matching based on generative adversarial networks[J]. Journal of Textile Research, 2021, 42(7): 164-168.
[11] MCAULEY J, TARGETT C, SHI Q, et al. Image-based recommendations on styles and substitutes[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2015: 43-52.
[12] VEIT A, KOVACS B, BELl S, et al. Learning visual clothing style with heterogeneous dyadic co-occurrences[C]// Proceedings of the IEEE International Conference on Computer Vision. Berli, Germany: Springer, 2015: 4642-4650.
[13] CHEN Long, HE Yuhang. Dress fashionably: learn fashion collocation with deep mixed-category metric learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 2103-2110.
[14] HE R, PACKER C, MCAULEY J. Learning compatibility across categories for heterogeneous item recommendation[C]// 2016 IEEE 16th International Conference on Data Mining. Barcelona, Spain:IEEE, 2017: 937-942.
[15] SHIH Y S, CHANG K Y, LIN H T, et al. Compatibility family learning for item recommendation and generation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 2403-2410.
[16] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[C]// IEEE Transactions on Pattern Analysis and Machine Intelligence. Piscataway, NJ: IEEE, 2018: 834-848.
[17] WANG Zheng, BAI Xiang, YE Mang, et al. Incremental deep hidden attribute learning[C]// Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: ACM Press, 2018: 72-80.
[18] 刘华玲, 马俊, 张国祥. 基于深度学习的内容推荐算法研究综述[J]. 计算机工程, 2021, 47(7): 1-12.
LIU Hualing, MA Jun, ZHANG Guoxiang. Review of studies on deep learning-based content recommendation algorithms[J]. Computer Engineering, 2021, 47(7): 1-12.
doi: 10.1063/1.31437
[19] HE R, LIN C, WANG J, et al. Sherlock: sparse hierarchical embeddings for visually-aware one-class collaborative filtering[C]// In International Joint Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 3740-3746.
[20] 张艺豪, 李梁, 赵清华, 等. 基于全局相似度的社交网络个性化推荐算法[J]. 计算机工程, 2018, 44(10): 204-208.
ZHANG Yihao, LI Liang, ZHAO Qinghua, et al. Personalized recommendation algorithm of social network based on global similarity[J]. Computer Engineering, 2018, 44(10): 204-208.
[21] BRILL M, ELKIND E, ENDRISS U, et al. Pairwise diffusion of preference rankings in social networks[C]// Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 130-136.
[22] 张卓, 丛洪莲, 蒋高明, 等. 基于交互式遗传算法的Polo衫快速款式推荐系统[J]. 纺织学报, 2021, 42(1): 138-144.
ZHANG Zhuo, CONG Honglian, JIANG Gaoming, et al. Polo shirt rapid style recommendation system based on interactive genetic algorithm[J]. Journal of Textile Research, 2021, 42(1): 138-144.
[23] 甘美辰, 李敏. 女装搭配推荐系统的设计与实现[J]. 纺织学报, 2020, 41(10): 122-131.
GAN Meichen, LI Min. Design and realization of a collocation recommendation system for women's clothing[J]. Journal of Textile Research, 2020, 41(10): 122-131.
[24] HE Xiangnan, ZHANG Hanwang, KAN Min-Yen, et al. Fast matrix factorization for online recommendation with implicit feedback[C]// Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2016: 549-558.
[25] KOREN Y, BELL R, VOLINSKY C. Matrix factorization techniques for recommender systems[J]. Computer, 2009, 42(8): 30-37.
[26] WANG Xiang, HE Xiangnan, WANG Meng, et al. Neural graph collaborative filtering[C]// Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2019: 165-174.
[27] RENDLE S, FREUDENTHALER C, GANTNER Z, et al. BPR: bayesian personalized ranking from implicit feedback[C]// Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence. Virginia, USA: AUAI Press, 2009: 452-461.
[28] HE R, MCAULEY J. VBPR: visual bayesian personalized ranking from implicit feedback[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1): 144-150.
[29] YU Wenhui, ZHANG Huidi, HE Xiangnan, et al. Aesthetic-based clothing recommendation[C]// Proceedings of the 2018 World Wide Web Conference. Lyon, France:IW3C2, 2018: 649-658.
[30] SONG Xuemeng, FENG Fuli, LIU Jinhuan, et al. Neurostylist: neural compatibility modeling for clothing matching[C]// Proceedings of the 25th ACM International Conference on Multimedia. New York, USA: ACM Press, 2017: 753-761.
[31] 贾俊杰, 刘鹏涛, 陈旺虎. 融合社交信息的矩阵分解改进推荐算法[J]. 计算机工程, 2021, 47(9): 97-105.
JIA Junjie, LIU Pengtao, CHEN Wanghu. Improved matrix factorization algorithm using social information for recommendation[J]. Computer Engineering, 2021, 47(9): 97-105.
[32] HU Yifan, KOREN Yehuda, VOLINSKY Chris. Collaborative filtering for implicit feedback data-sets[C]// Proceedings of the 2008 Eighth IEEE International Conference on Data Mining. New York, USA: IEEE Computer Society, 2008: 263-272.
[33] PAN Rong, ZHOU Yunhong, CAO Bin, et al. One-class collaborative filtering[C]// Proceedings of the 2008 Eighth IEEE International Conference on Data Mining. New York, USA: IEEE Computer Society, 2008: 502-511.
[34] REN Shaoqing, HE Kaiming, GIRSHICK Ross. Faster r-cnn: towards real-time object detection with region proposal networks[J]. IEEE Trans Pattern Anal Mach Intell, 2017, 39(6): 1137-1149.
doi: 10.1109/TPAMI.2016.2577031
[35] 孙洁, 丁笑君, 杜磊, 等. 基于卷积神经网络的织物图像特征提取与检索研究进展[J]. 纺织学报, 2019, 40(12): 146-151.
SUN Jie, DING Xiaojun, DU Lei, et al. Research progress of fabric image feature extraction and retrieval based on convolutional neural network[J]. Journal of Textile Research, 2019, 40(12): 146-151.
[36] 王柳程, 欧阳城添, 梁文. 基于改进特征金字塔网络的人体姿态估计[J]. 计算机工程, 2021, 47(8): 251-259,270.
WANG Liucheng, OUYANG Chengtian, LIANG Wen. Human pose estimation based on improved pyramid feature network[J]. Computer Engineering, 2021, 47(8): 251-259,270.
[37] 音松, 陈雪云, 贝学宇. 改进Mask RCNN算法及其在行人实例分割中的应用[J]. 计算机工程, 2021, 47(6): 271-276,283.
YIN Song, CHEN Xueyun, BEI Xueyu. Improved mask rcnn algorithm and its application in pedestrian instance segmentation[J]. Computer Engineering, 2021, 47(6): 271-276,283.
doi: 10.1063/1.31441
[38] LIN T, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE, 2017: 936-944.
[39] KIM Y. Convolutional neural networks for sentence classification[C]// Proceedings of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2014. DOI: 10.48550/arXiv.1408.5882.
doi: 10.48550/arXiv.1408.5882
[40] 喻靖民, 向凌云, 曾道建. 基于Word2vec的自然语言隐写分析方法[J]. 计算机工程, 2019, 45(3): 309-314.
YU Jingmin, XIANG Lingyun, ZENG Daojian. Natural language steganalysis method based on Word2vec[J]. Computer Engineering, 2019, 45(3): 309-314.
[41] HU Zhiting, MA Xuezhe, LIU Zhengzhong. Harnessing deep neural networks with logic rules[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2016: 2410-2420.
[42] SEVERYN A, MOSCHITTI A. Twitter sentiment analysis with deep convolutional neural networks[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2015: 959-962.
[43] ASAHARA M. NWJC2Vec: word embedding dataset from 'ninjal web japanese corpus'[J]. Terminology, 2018, 24(1): 7-22.
[44] KINGMA D P, BA J. Adam: a method for stochastic optimization[J]. Computer Science, 2014. DOI: 10.48550/arxiv.1412.6980.
doi: 10.48550/arxiv.1412.6980
[45] BRADLEY P. The use of the area under the ROC curve in the evaluation of machine learning algorithms[J]. Pattern Recognition, 1997, 30(7): 1145-1159.
doi: 10.1016/S0031-3203(96)00142-2
[46] SONG Xuemeng, HAN Xianjing, LI Yunkai, et al. GP-BPR: personalized compatibility modeling for clothing matching[C]// Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 320-328.
[47] HAN Xintong, WU Zuxuan, JIANG Yugang, et al. Learning fashion compatibility with bidirectional LSTMs[C]// Proceedings of the 25th ACM International Conference on Multimedia. New York, USA: ACM Press, 2017: 1078-1086.
[48] HE Xiangnan, LIAO Lizi, ZHANG Hanwang, et al. Neural collaborative filtering[C]// Proceedings of the 26th International Conference on World Wide Web. Republic and Canton of Geneva, CHE:IW3C2, 2017: 173-182.
[49] JIANG Lu, YU S I, MENG Deyu, et al. Fast and accurate content-based semantic search in 100m internet videos[C]// Proceedings of the 23rd ACM International Conference on Multimedia. New York, USA: ACM Press, 2015: 49-58.
[50] YIN Hongzhi, CHEN Hongxu, SUN Xiaoshuai, et al. SPTF: a scalable probabilistic tensor factorization model for semantic-aware behavior prediction[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(7): 1763-1777.
doi: 10.1109/TKDE.2013.168
[1] MA Chuanxu, ZHANG Ning, PAN Ruru. Detection of cheese yarn bobbin varieties based on support vector machine [J]. Journal of Textile Research, 2023, 44(01): 194-200.
[2] XU Zengbo, ZHANG Ling, ZHANG Yanhong, CHEN Guiqing. Research on clothing collar types based on complex network extraction and support vector machine classification [J]. Journal of Textile Research, 2021, 42(06): 146-152.
[3] ZHENG Xiaoping, LIU Chengxia. Objective evaluation of woven garment bagging performance by laser measurement [J]. Journal of Textile Research, 2021, 42(05): 143-149.
[4] XIA Haibang, HUANG Hongyun, DING Zuohua. Clothing comfort evaluation based on transfer learning and support vector machine [J]. Journal of Textile Research, 2020, 41(06): 125-131.
[5] WANG Hui, SUN Jie, DING Xiaojun, LONG Ying, ZOU Fengyuan. Comparison of feature extracting and matching methods for fabric patterns [J]. Journal of Textile Research, 2020, 41(04): 45-50.
[6] ZHU Hao, DING Hui, SHANG Yuanyuan, SHAO Zhuhong. Defect detection algorithm for multiple texture hierarchical fusion fabric [J]. Journal of Textile Research, 2019, 40(06): 117-124.
[7] XING Wenyu, DENG Na, XIN Binjie, YU Chen. Identification of wool and cashmere based on multi-feature fusion image analysis technology [J]. Journal of Textile Research, 2019, 40(03): 146-152.
[8] . Fabric defect inspection based on modified discriminant complete local binary pattern and lattice segmentation [J]. Journal of Textile Research, 2018, 39(09): 57-64.
[9] . Global and local diversity features-fused colorimetry index testing and evaluation of colored spun yarns [J]. JOURNAL OF TEXTILE RESEARCH, 2018, 39(02): 157-164.
[10] . Fast fabric defect detection algorithm based on integral image [J]. JOURNAL OF TEXTILE RESEARCH, 2016, 37(11): 141-147.
[11] . Lace retrieval method based on improved texture feature [J]. JOURNAL OF TEXTILE RESEARCH, 2016, 37(06): 142-154.
[12] . Automatic recognition and positioning method of fabric slitting zone using one-dimensional Fourier transform [J]. JOURNAL OF TEXTILE RESEARCH, 2016, 37(01): 147-151.
[13] . Algorithm for pattern detection of cotton foreign fibers based on cluster statistic analysis [J]. Journal of Textile Research, 2015, 36(03): 135-139.
[14] . Defect detection of plain weave based on visual saliency mechanism [J]. JOURNAL OF TEXTILE RESEARCH, 2014, 35(4): 56-0.
[15] . Fabric defect detection of statistic aberration feature based on GMRF model [J]. JOURNAL OF TEXTILE RESEARCH, 2013, 34(4): 137-142.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
京ICP备14006648号
Copyright 2021 © Editorial office of Journal of Textile Research
Supported by Beijing Magtech Co.Ltd