基于隐层傅里叶卷积的非平稳纹理合成方法

doi:10.3969/j.issn.1000-5641.2024.02.013

摘要/Abstract

摘要：

随着深度学习在计算机视觉领域取得的巨大成功, 基于示例的纹理合成研究得到了长足的发展. 当下主流纹理合成模型往往采用神经网络方法, 其通常包含卷积层和上采样层、下采样层等局部组件, 并不适用于捕捉非平稳纹理中的不规则结构特征. 受频率域与空间域的对偶性质的启发, 提出了一种基于隐层傅里叶卷积的非平稳纹理合成方法. 该方法以生成对抗网络为基础架构, 沿着隐层通道进行特征拆分, 搭建图像域局部分支和频率域全局分支, 进而兼顾视觉感知和结构信息. 实验表明, 该方法能够处理结构上极具挑战的非平稳纹理样本, 相较于目前最优方法而言, 在大尺度结构的学习与扩展上取得了更好的效果.

关键词: 纹理合成, 非平稳纹理, 傅里叶卷积, 生成对抗网络

Abstract:

The remarkable achievements of deep learning in computer vision have led to significant development in example-based texture synthesis. The texture synthesis model using neural networks mainly includes local components, such as convolution and up/down sampling, which is unsuitable for capturing irregular structural attributes in non-stationary textures. Inspired by the frequency and space domain duality, a non-stationary texture synthesis method based on hidden layer Fourier convolution is proposed in this study. The proposed method uses the generative adversarial network as the basic architecture, performs feature splitting along the channel in the hidden layer, and builds a local branch in the image domain and a global branch in the frequency domain to consider visual perception and structural information. Experimental results show that this method can handle structurally challenging non-stationary texture exemplars. Compared with state-of-the-art methods, the method yielded better results in the learning and expansion of large-scale structures.

Key words: texture synthesis, non-stationary texture, Fourier convolution, generative adversarial network

中图分类号:

TP183

何鑫鑫, 宋海川. 基于隐层傅里叶卷积的非平稳纹理合成方法[J]. 华东师范大学学报（自然科学版）, 2024, 2024(2): 119-130.

Xinxin HE, Haichuan SONG. Hidden layer Fourier convolution for non-stationary texture synthesis[J]. Journal of East China Normal University(Natural Science), 2024, 2024(2): 119-130.

图/表 7

图1

图2

图3

图4

表1

图5

图6

参考文献 36

1	ZHOU Y, ZHU Z, BAI X, et al.. Non-stationary texture synthesis by adversarial expansion. ACM Transactions on Graphics, 2018, 37 (4): 49.
2	GATYS L A, ECKER A S, BETHGE M. Texture synthesis using convolutional neural networks [C]// Proceedings of the 28th International Conference on Neural Information Processing Systems. 2015: 262-270.
3	LI Y J, FANG C, YANG J M, et al. Universal style transfer via feature transforms [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 385-395.
4	MARDANI M, LIU G, DUNDAR A, et al.. Neural FFTs for universal texture image synthesis. Advances in Neural Information Processing Systems, 2020, 33, 14081- 14092.
5	HEITZ E, VANHOEY K, CHAMBON T, et al. A sliced wasserstein loss for neural texture synthesis [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 9412-9420.
6	ABDERRAZAK A M, HACHOUF F. Texture synthesis using improved transfer learning [C]// 2022 4th International Conference on Pattern Analysis and Intelligent Systems. DOI: 10.1109/PAIS56586.2022.9946881.
7	LIU G, GOUSSEAU Y, XIA G S. Texture synthesis through convolutional neural networks and spectrum constraints [C]// Proceedings of the 2016 23rd International Conference on Pattern Recognition. 2016: 3234-3239.
8	LI Y, FANG C, YANG J, et al. Diversified texture synthesis with feed-forward networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 3920-3928.
9	XIE S S, QIAN W, NIE R, et al.. GAGCN: Generative adversarial graph convolutional network for non-homogeneous texture extension synthesis. IET Image Processing, 2023, 17, 1603- 1614.
10	GRANOT N, FEINSTEIN B, SHOCHER A, et al. Drop the GAN: In defense of patches nearest neighbors as single image generative models [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 13460-13469.
11	ELNEKAVE A, WEISS Y. Generating natural images with direct patch distributions matching [C]// Computer Vision–ECCV 2022: 17th European Conference. 2022: 544-560.
12	WANG Z M, LI M H, XIA G S.. Conditional generative ConvNets for exemplar-based texture synthesis. IEEE Transactions on Image Processing, 2021, 30, 2461- 2475.
13	HOUDARD A, LECLAIRE A, PAPADAKIS N, et al.. A generative model for texture synthesis based on optimal transport between feature distributions. Journal of Mathematical Imaging and Vision, 2023, 65 (1): 4- 28.
14	SHOCHER A, BAGON S, ISOLA P, et al. InGAN: Capturing and remapping the “DNA” of a natural image [C]// Proceedings of the IEEE International Conference on Computer Vision. 2019: 4492-4501.
15	FRÜHSTÜCK A, ALHASHIM I, WONKA P.. TileGAN: Synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics, 2019, 38 (4): 58.
16	HOUDARD A, LECLAIRE A, PAPADAKIS N, et al. Wasserstein generative models for patch-based texture synthesis [C]// Scale Space and Variational Methods in Computer Vision: 8th International Conference. 2021: 269-280.
17	DARZI A, LANG I, TAKLIKAR A, et al.. Co-occurrence based texture synthesis. Computational Visual Media, 2022, (8): 289- 302.
18	LIN J, SHARMA G, PAPPAS T N.. Toward universal texture synthesis by combining texton broadcasting with noise injection in StyleGAN-2. e-Prime-Advances in Electrical Engineering, Electronics and Energy, 2023, (3): 100092.
19	SHAHAM T R, DEKEL T, MICHAELI T. SinGAN: Learning a generative model from a single natural image [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4570-4580.
20	EFROS A A, LEUNG T K. Texture synthesis by non-parametric sampling [C]// Proceedings of the seventh IEEE International Conference on Computer Vision. 1999: 1033-1038.
21	EFROS A A, FREEMAN W T. Image quilting for texture synthesis and transfer [C]// Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 2001: 341-346.
22	黄志勇, 何发智, 张胜龙, 等.. 基于随机查找的并行大规模纹理合成. 计算机辅助设计与图形学学报, 2011, 23 (6): 1091- 1098.
23	张伟伟, 何凯, 孟春芝.. 自适应选取样本块大小的纹理合成方法. 计算机工程与应用, 2012, 48 (17): 170- 173.
24	KWATRA V, SCHÖDL A, ESSA I, et al.. Graphcut textures: Image and video synthesis using graph cuts. ACM Transactions on Graphics, 2003, 22 (3): 277- 286.
25	孙劲光, 刘双九.. 块尺寸自适应的 Tile 纹理合成算法. 计算机工程与应用, 2016, 52 (11): 164- 168.
26	KWATRA V, ESSA I, BOBICK A, et al.. Texture optimization for example-based synthesis. ACM Transactions on Graphics, 2005, 24 (3): 795- 802.
27	ROSENBERGER A, COHEN-OR D, LISCHINSKI D.. Layered shape synthesis: Automatic generation of control maps for non-stationary textures. ACM Transactions on Graphics, 2009, 28 (5): 107.
28	KASPAR A, NEUBERT B, LISCHINSKI D, et al. Self tuning texture optimization [C]// Computer Graphics Forum. 2015: 349-359.
29	LEFEBVRE S, HOPPE H.. Appearance-space texture synthesis. ACM Transactions on Graphics, 2006, 25 (3): 541- 548.
30	HERTZMANN A, JACOBS C E, OLIVER N, et al. Image analogies [C]// Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 2001: 327-340.
31	BERGMANN U, JETCHEV N, VOLLGRAF R. Learning texture manifolds with the Periodic Spatial GAN [C]// Proceedings of the 34th International Conference on Machine Learning. 2017: 469-477.
32	LI C, WAND M. Precomputed real-time texture synthesis with Markovian generative adversarial networks [C]// Computer Vision–ECCV 2016: 14th European Conference. 2016: 702-716.
33	JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution [C]// Computer Vision–ECCV 2016: 14th European Conference. 2016: 694-711.
34	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks [C]// Proceedings of the IEEE International Conference on Computer Vision. 2017: 2223-2232.
35	SUVOROV R, LOGACHEVA E, MASHIKHIN A, et al. Resolution-robust large mask inpainting with Fourier convolutions [C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 2149-2159.
36	CHI L, JIANG B, MU Y.. Fast Fourier convolution. Advances in Neural Information Processing Systems, 2020, 33, 4479- 4488.

方法	模型参数量	模型计算力
基线方法^[1]	$ {\mathit{C}}^{2}{\mathit{K}}^{2} $	$ {\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{\mathit{l}\to \mathit{l}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{\mathit{l}\to {g}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{{g}\to \mathit{l}} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} $	$ \dfrac{1}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法$ {\mathit{F}}^{{g}\to {g}} $	$ \dfrac{1}{2}{\mathit{C}}^{2} $	$ \dfrac{21}{64}{\mathit{C}}^{2}\mathit{H}\mathit{W} $
本文所提出的方法	$ \dfrac{3}{4}{\mathit{C}}^{2}{\mathit{K}}^{2} + \dfrac{1}{2}{\mathit{C}}^{2} $	$ \dfrac{3}{4}{\mathit{C}}^{2}{\mathit{K}}^{2}\mathit{H}\mathit{W} + \dfrac{21}{64}{\mathit{C}}^{2}\mathit{H}\mathit{W} $

[1]	步一凡, 王晓玲, 贺珂珂, 卢兴见, 王文萱. 面向身份相互关系一致性的人脸去识别化方法[J]. 华东师范大学学报（自然科学版）, 2023, 2023(6): 49-60.
[2]	王泽华, 高岩, 陈敏刚. 基于遥感图像的虚拟地形仿真建模方法[J]. 华东师范大学学报（自然科学版）, 2023, 2023(2): 82-94.