Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in speci...Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environments.展开更多
For swarm robots moving in a harsh or uncharted outdoor environment without GPS guidance and global communication,algorithms that rely on global-based information are infeasible.Typically,traditional gene regulatory n...For swarm robots moving in a harsh or uncharted outdoor environment without GPS guidance and global communication,algorithms that rely on global-based information are infeasible.Typically,traditional gene regulatory networks(GRNs)that achieve superior performance in forming trapping pattern towards targets require accurate global positional information to guide swarm robots.This article presents a gene regulatory network with Self-organized grouping and entrapping method for swarms(SUNDER-GRN)to achieve adequate trapping performance with a large-scale swarm in a confined multitarget environment with access to only local information.A hierarchical self-organized grouping method(HSG)is proposed to structure subswarms in a distributed way.In addition,a modified distributed controller,with a relative coordinate system that is established to relieve the need for global information,is leveraged to facilitate subswarms entrapment toward different targets,thus improving the global multi-target entrapping performance.The results demonstrate the superiority of SUNDERGRN in the performance of structuring subswarms and entrapping 10 targets with 200 robots in an environment confined by obstacles and with only local information accessible.展开更多
The occurrence of social security events is uncertain, and the distribution characteristics are highly complex due to a variety of external factors, posing challenges to their rapid and effective handling. The scienti...The occurrence of social security events is uncertain, and the distribution characteristics are highly complex due to a variety of external factors, posing challenges to their rapid and effective handling. The scientific and reasonable requirement evaluation of the emergency force to deal with social security events is very urgent. Based on data analysis, this paper uses the neural network, operational research, modelling and simulation to predict and analyze social security events, studies the usage rule of emergency force and deployment algorithm, and conducts simulation experiments to evaluate and compare the different force deployment schemes for selection.展开更多
基金supported by the National Key R&D Program of China(No.2021YFB0301200)National Natural Science Foundation of China(No.62025208).
文摘Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environments.
基金supported in part by National Key R&D Program of China(Grant Nos.2021ZD0111501,2021ZD0111502)the Key Laboratory of Digital Signal and Image Processing of Guangdong Province+8 种基金the Key Laboratory of Intelligent Manufacturing Technology(Shantou University)Ministry of Education,the Science and Technology Planning Project of Guangdong Province of China(Grant No.180917144960530)the Project of Educational Commission of Guangdong Province of China(Grant No.2017KZDXM032)the State Key Lab of Digital Manufacturing Equipment&Technology(grant number DMETKF2019020)National Natural Science Foundation of China(Grant Nos.62176147,62002369)STU Scientific Research Foundation for Talents(Grant No.NTF21001)Science and Technology Planning Project of Guangdong Province of China(Grant Nos.2019A050520001,2021A0505030072,2022A1515110660)Science and Technology Special Funds Project of Guangdong Province of China(Grant Nos.STKJ2021176,STKJ2021019)Guangdong Special Support Program for Outstanding Talents(Grant No.2021JC06X549)。
文摘For swarm robots moving in a harsh or uncharted outdoor environment without GPS guidance and global communication,algorithms that rely on global-based information are infeasible.Typically,traditional gene regulatory networks(GRNs)that achieve superior performance in forming trapping pattern towards targets require accurate global positional information to guide swarm robots.This article presents a gene regulatory network with Self-organized grouping and entrapping method for swarms(SUNDER-GRN)to achieve adequate trapping performance with a large-scale swarm in a confined multitarget environment with access to only local information.A hierarchical self-organized grouping method(HSG)is proposed to structure subswarms in a distributed way.In addition,a modified distributed controller,with a relative coordinate system that is established to relieve the need for global information,is leveraged to facilitate subswarms entrapment toward different targets,thus improving the global multi-target entrapping performance.The results demonstrate the superiority of SUNDERGRN in the performance of structuring subswarms and entrapping 10 targets with 200 robots in an environment confined by obstacles and with only local information accessible.
文摘The occurrence of social security events is uncertain, and the distribution characteristics are highly complex due to a variety of external factors, posing challenges to their rapid and effective handling. The scientific and reasonable requirement evaluation of the emergency force to deal with social security events is very urgent. Based on data analysis, this paper uses the neural network, operational research, modelling and simulation to predict and analyze social security events, studies the usage rule of emergency force and deployment algorithm, and conducts simulation experiments to evaluate and compare the different force deployment schemes for selection.
