Hidden layer Fourier convolution for non-stationary texture synthesis

doi:10.3969/j.issn.1000-5641.2024.02.013

Abstract

Abstract:

The remarkable achievements of deep learning in computer vision have led to significant development in example-based texture synthesis. The texture synthesis model using neural networks mainly includes local components, such as convolution and up/down sampling, which is unsuitable for capturing irregular structural attributes in non-stationary textures. Inspired by the frequency and space domain duality, a non-stationary texture synthesis method based on hidden layer Fourier convolution is proposed in this study. The proposed method uses the generative adversarial network as the basic architecture, performs feature splitting along the channel in the hidden layer, and builds a local branch in the image domain and a global branch in the frequency domain to consider visual perception and structural information. Experimental results show that this method can handle structurally challenging non-stationary texture exemplars. Compared with state-of-the-art methods, the method yielded better results in the learning and expansion of large-scale structures.

Key words: texture synthesis, non-stationary texture, Fourier convolution, generative adversarial network

CLC Number:

TP183

Xinxin HE, Haichuan SONG. Hidden layer Fourier convolution for non-stationary texture synthesis[J]. Journal of East China Normal University(Natural Science), 2024, 2024(2): 119-130.

Figures/Tables 7

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Fig.5

Fig.6

References 36

1	ZHOU Y, ZHU Z, BAI X, et al.. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics, 2018, 37 (4): 49.
2	GATYS L A, ECKER A S, BETHGE M. Texture synthesis using convolutional neural networks [C]// Proceedings of the 28th International Conference on Neural Information Processing Systems. 2015: 262-270.
3	LI Y J, FANG C, YANG J M, et al. Universal style transfer via feature transforms [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 385-395.
4	MARDANI M, LIU G, DUNDAR A, et al.. Neural FFTs for universal texture image synthesis. Advances in Neural Information Processing Systems, 2020, 33, 14081- 14092.
5	HEITZ E, VANHOEY K, CHAMBON T, et al. A sliced wasserstein loss for neural texture synthesis [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 9412-9420.
6	ABDERRAZAK A M, HACHOUF F. Texture synthesis using improved transfer learning [C]// 2022 4th International Conference on Pattern Analysis and Intelligent Systems. DOI: 10.1109/PAIS56586.2022.9946881.
7	LIU G, GOUSSEAU Y, XIA G S. Texture synthesis through convolutional neural networks and spectrum constraints [C]// Proceedings of the 2016 23rd International Conference on Pattern Recognition. 2016: 3234-3239.
8	LI Y, FANG C, YANG J, et al. Diversified texture synthesis with feed-forward networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 3920-3928.
9	XIE S S, QIAN W, NIE R, et al.. GAGCN: Generative adversarial graph convolutional network for non-homogeneous texture extension synthesis. IET Image Processing, 2023, 17, 1603- 1614.
10	GRANOT N, FEINSTEIN B, SHOCHER A, et al. Drop the GAN: In defense of patches nearest neighbors as single image generative models [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 13460-13469.
11	ELNEKAVE A, WEISS Y. Generating natural images with direct patch distributions matching [C]// Computer Vision–ECCV 2022: 17th European Conference. 2022: 544-560.
12	WANG Z M, LI M H, XIA G S.. Conditional generative ConvNets for exemplar-based texture synthesis. IEEE Transactions on Image Processing, 2021, 30, 2461- 2475.
13	HOUDARD A, LECLAIRE A, PAPADAKIS N, et al.. A generative model for texture synthesis based on optimal transport between feature distributions. Journal of Mathematical Imaging and Vision, 2023, 65 (1): 4- 28.
14	SHOCHER A, BAGON S, ISOLA P, et al. InGAN: Capturing and remapping the “DNA” of a natural image [C]// Proceedings of the IEEE International Conference on Computer Vision. 2019: 4492-4501.
15	FRÜHSTÜCK A, ALHASHIM I, WONKA P.. TileGAN: Synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics, 2019, 38 (4): 58.
16	HOUDARD A, LECLAIRE A, PAPADAKIS N, et al. Wasserstein generative models for patch-based texture synthesis [C]// Scale Space and Variational Methods in Computer Vision: 8th International Conference. 2021: 269-280.
17	DARZI A, LANG I, TAKLIKAR A, et al.. Co-occurrence based texture synthesis. Computational Visual Media, 2022, (8): 289- 302.
18	LIN J, SHARMA G, PAPPAS T N.. Toward universal texture synthesis by combining texton broadcasting with noise injection in StyleGAN-2. e-Prime-Advances in Electrical Engineering, Electronics and Energy, 2023, (3): 100092.
19	SHAHAM T R, DEKEL T, MICHAELI T. SinGAN: Learning a generative model from a single natural image [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4570-4580.
20	EFROS A A, LEUNG T K. Texture synthesis by non-parametric sampling [C]// Proceedings of the seventh IEEE International Conference on Computer Vision. 1999: 1033-1038.
21	EFROS A A, FREEMAN W T. Image quilting for texture synthesis and transfer [C]// Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 2001: 341-346.
22	黄志勇, 何发智, 张胜龙, 等.. 基于随机查找的并行大规模纹理合成. 计算机辅助设计与图形学学报, 2011, 23 (6): 1091- 1098.
23	张伟伟, 何凯, 孟春芝.. 自适应选取样本块大小的纹理合成方法. 计算机工程与应用, 2012, 48 (17): 170- 173.
24	KWATRA V, SCHÖDL A, ESSA I, et al.. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics, 2003, 22 (3): 277- 286.
25	孙劲光, 刘双九.. 块尺寸自适应的 Tile 纹理合成算法. 计算机工程与应用, 2016, 52 (11): 164- 168.
26	KWATRA V, ESSA I, BOBICK A, et al.. Texture optimization for example-based synthesis. ACM Transactions on Graphics, 2005, 24 (3): 795- 802.
27	ROSENBERGER A, COHEN-OR D, LISCHINSKI D.. Layered shape synthesis: Automatic generation of control maps for non-stationary textures. ACM Transactions on Graphics, 2009, 28 (5): 107.
28	KASPAR A, NEUBERT B, LISCHINSKI D, et al. Self tuning texture optimization [C]// Computer Graphics Forum. 2015: 349-359.
29	LEFEBVRE S, HOPPE H.. Appearance-space texture synthesis. ACM Transactions on Graphics, 2006, 25 (3): 541- 548.
30	HERTZMANN A, JACOBS C E, OLIVER N, et al. Image analogies [C]// Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 2001: 327-340.
31	BERGMANN U, JETCHEV N, VOLLGRAF R. Learning texture manifolds with the Periodic Spatial GAN [C]// Proceedings of the 34th International Conference on Machine Learning. 2017: 469-477.
32	LI C, WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks [C]// Computer Vision–ECCV 2016: 14th European Conference. 2016: 702-716.
33	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution [C]// Computer Vision–ECCV 2016: 14th European Conference. 2016: 694-711.
34	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 2223-2232.
35	SUVOROV R, LOGACHEVA E, MASHIKHIN A, et al. Resolution-robust large mask inpainting with Fourier convolutions [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 2149-2159.
36	CHI L, JIANG B, MU Y.. Fast Fourier convolution. Advances in Neural Information Processing Systems, 2020, 33, 4479- 4488.

方法	模型参数量	模型计算力
基线方法^[1]	$ {\mathit{C}}^{2}{\mathit{K}}^{2} $	$ {\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{\mathit{l}\to \mathit{l}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{\mathit{l}\to {g}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{{g}\to \mathit{l}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{{g}\to {g}} $	$ \dfrac{1}{2}{\mathit{C}}^{2} $	$ \dfrac{21}{64}{\mathit{C}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法	$ \dfrac{3}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} + \dfrac{1}{2}{\mathit{C}}^{2} $	$ \dfrac{3}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} + \dfrac{21}{64}{\mathit{C}}^{2}\mathit{H}\mathit{W} $

[1]	Yifan BU, Xiaoling WANG, Keke HE, Xingjian LU, Wenxuan WANG. Towards an identity inter-relationship-consistent face de-identification method [J]. Journal of East China Normal University(Natural Science), 2023, 2023(6): 49-60.
[2]	Zehua WANG, Yan GAO, Mingang CHEN. A landscape simulation modeling method based on remote sensing images [J]. Journal of East China Normal University(Natural Science), 2023, 2023(2): 82-94.