摘要
以众多遥感卫星的发射与行业应用系统为代表,遥感应用技术在近年间取得了快速发展,国产高分系列卫星等平台所提供的数据越来越丰富,以遥感共性产品作为信息源的应用系统也已经覆盖了多个行业。然而这两者之间的桥梁,即由国产卫星数据生产标准化系列遥感共性产品的能力,仍显得相对薄弱,极大限制了国产遥感数据的使用率与影响力。这一能力的改善需要从产品体系、产品算法与生产系统等多方面共同推进。本文从高分遥感共性产品生产系统的角度,针对系统所面对的来自数据集成、算法集成、生产编排与云计算等方面的诸多挑战,提出了一套完整的系统关键设计。系统在容器化集成运行的基础上,涵盖了集群、数据、算法、权限、参数、工作流、生产等一系列关键环节的关键技术。系统研发与生产实践表明,本系统可以流畅的集成各类共性产品算法,实现稳定高效的业务化生产平台,助力于国产卫星产品服务能力的有效提高。
A fast development in remote sensing science and technologies has been witnessed in recent years with the launch of various remote sensing satellites and the establishment of numerous industrial application systems.Satellite series,like GaoFen,has pushed the richness of data to a new level.Moreover,quantitative product-driven application systems have become increasingly influential in many disciplines.By contrast,the bridge between data and application,namely,the production capability of quantitative products,seems weak,greatly limiting the usage and influence of GaoFen data.The improvement of production capability comes from multiple aspects,including product hierarchy,algorithm model,and production system.In this study,we propose,from the production system perspective,an integrated system design that responds to the four main challenges of the system:uniform data access,heterogeneous algorithm integration,layered workflow orchestration,and cloud infrastructure adaptation.The system is based on algorithm containerization,where executable algorithms and all their dependencies are encapsulated.Thus,it can run uniformly and consistently on different infrastructures without worrying about the complexity caused by deployment.This scenario helps the system to manage diverse remote sensing algorithms uniformly.We employ the Kubernetes container orchestration platform to automate the execution,scaling,and management of containerized algorithms.A containerized cluster consists of multiple master nodes,many computing nodes,and multiple data centers.Multiple algorithm repositories are constructed to support the system and cope with the high computing and data throughput density of remote sensing algorithms.Each algorithm repository is further divided into several subrepositories to improve load balancing.User-defined role-based access control for algorithms is set up to protect the intellectual properties of algorithm owners.A recommended algorithm image architecture is introduced to standardize algorithm encapsulation.A set of nine properties are abstracted to describe uniformly any data entity parameter of an algorithm.This approach ensures that suitable input data can be found for user-uploaded algorithms to run in the system.For data visualization and quantitative computing scenarios,a multiscenario data organization strategy is proposed to avoid excessive data operations,such as projection transform or subdivision.The business logic of the system,from user order creation to product calculation,is detailed for clear implementation.The production sometimes involves workflow batches.We propose a stratified workflow aggregation strategy to optimize workflow execution.The system has been used for large-scale production of various GF quantitative remote sensing products,including surface reflectance product,normalized difference vegetation index product,leaf area index product,and surface albedo product.These products fully cover China’s area for eight successive years from 2013 to 2020,with quantities of more than five million and storages of nearly 300 TB.The proposed system completes the production task smoothly and efficiently.During the routine support for many large-scale production tasks,each part of the system performed consistently with the system design proposed in this study,demonstrating that the study can help build a stable and efficient quantitative remote sensing production system on cloud-native infrastructures.
作者
张正
李宏益
胡昌苗
唐娉
ZHANG Zheng;LI Hongyi;HU Changmiao;TANG Ping(Aerospace Information Research Institute,Chinese Academy of Sciences,Beijing 100094,China)
出处
《遥感学报》
EI
CSCD
北大核心
2023年第3期651-664,共14页
NATIONAL REMOTE SENSING BULLETIN
基金
国家重点研发计划(编号:2018YFA0605500)
国家自然科学基金青年科学基金(编号:41701399)
高分辨率对地观测系统重大专项(编号:21-Y20B02-9003-19/22)。
关键词
生产系统
共性产品
算法集成
容器
云计算
系统架构
工作流
production system
quantitative product
algorithm integration
container
cloud computing
system architecture
workflow
