Research Report on AI Foundation Models and Their Applications in Automotive Field, 2024-2025
상품코드:1660087
리서치사:ResearchInChina
발행일:2025년 02월
페이지 정보:영문 340 Pages
라이선스 & 가격 (부가세 별도)
한글목차
추론 능력이 기반 모델의 성능을 밀어 올립니다.
2024년 후반 이후 중국 내외의 기반 모델 기업은 추론 모델을 발표하고 Chain-of-Thought(CoT)와 같은 추론 프레임워크를 사용하여 기반 모델이 복잡한 작업을 처리하고 독립적으로 의사결정을 할 수 있는 능력을 강화하고 있습니다.
추론 모델의 집중적인 릴리스는 복잡한 시나리오를 처리하기 위한 기반 모델의 능력을 강화하고 Agent 용도에 대한 기초를 구축하는 것을 목표로 합니다. 예를 들면, 복잡한 시맨틱스에 있어서의 콕핏 어시스턴트의 의도 인식의 강화나, 자동 운전 계획·결정에 있어서의 시공간 예측의 정밀도 향상 등입니다.
2024년 자동차에 탑재된 주류 기반 모델의 추론 기술은 주로 CoT와 그 변종, 예를 들어 ToT(Tree-of-Thought), GoT(Graph-of-Thought), FoT(Forest-of-Thought)를 중심으로 전개되어 생성 모델(예를 들면 확산 모델), 지식 그래프, 인과 추론 모델, 누적 추론 및 다중 모드 추론 체인과 결합되었습니다.
예를 들어, Geely가 제안한 Modularized Thinking Language Model(MeTHanol)은 기반 모델이 인간의 사고를 합성하여 LLM의 숨겨진 레이어를 감독할 수 있게 하고, 인간과 같은 사고 행동을 생성해, 일상 대화나 개인화된 프롬프트에 적응하는 것에 의해 대규모
2025년 추론기술의 초점은 멀티모달 추론으로 전환됩니다. 일반적인 트레이닝 기술은 명령 미세 조정, 멀티모달 컨텍스트 학습, 멀티모달 CoT(M-CoT)를 포함하며, 많은 경우 멀티모달 융합 정렬과 LLM 추론 기술을 결합하여 가능합니다.
설명 가능성은 AI와 사용자의 신뢰 관계를 교차시킵니다.
사용자는 AI의 "유용성"을 경험하기 전에 AI를 신뢰해야합니다. 2025년 AI 시스템의 설명 가능성은 자동차 AI 사용자를 늘리는 데 중요한 요소입니다. 이 과제는 긴 CoT를 입증함으로써 해결할 수 있습니다.
AI 시스템의 설명 가능성은 데이터 설명 가능성, 모델 설명 가능성, 사후 설명 가능성의 세 가지 수준에서 달성될 수 있습니다.
Li Auto의 경우 L3 자율주행은 'AI 추론 시각화 기술'을 사용하여 엔드 투 엔드 VLM 모델의 사고 프로세스를 직관적으로 제시하고, 물리 세계의 지각 입력에서 기반 모델에 의해 출력되는 운전 판단까지의 전체 프로세스를 커버하고, 지능형 드라이빙 시스템에 대한 사용자의 신뢰를 높이고 있습니다.
Li Auto의 "AI 추론 시각화 기술"에서는
주의 시스템은 차량이 인식한 교통 및 환경 정보를 표시하고, 실시간 비디오 스트림에서 교통 참가자의 행동을 평가하며, 히트맵에서 평가 대상을 표시합니다.
엔드 투 엔드(E2E) 모델은 주행 궤적 출력 뒤에 있는 사고 과정을 보여줍니다. 이 모델은 다양한 주행 궤적에 대해 생각하고 10개의 출력 후보 결과를 제시하며 궁극적으로 가장 가능성이 높은 출력 결과를 주행 궤적으로 채택합니다.
시각 언어 모델(VLM)은 지각, 추론 및 의사 결정 과정을 대화식으로 표시합니다.
다양한 추론 모델의 상호 작용 인터페이스는 유사하게 추론 프로세스를 분해하기 위해 긴 CoT를 채택합니다. 예를 들어, DeepSeek R1에서는 사용자와의 대화에서 먼저 CoT가 각 노드에서 결정을 제시한 다음 자연어로 설명합니다.
또한 Zhipu의 GLM-Zero-Preview, Alibaba의 QwQ-32B-Preview, Skywork 4.0 o1 등 대부분의 추론 모델은 긴 CoT 추론 프로세스의 시연을 지원합니다.
이 보고서는 중국의 자동차 산업에 대해 조사했으며, AI 기반 모델의 개요, 유형, 공통 기술, 기업, 자동차에의 적용 사례 등의 정보를 제공합니다.
목차
제1장 AI 기반 모델 개요
AI 모델의 소개
기반 모델의 소개
제2장 다른 유형의 AI 기반 모델 분석
대규모 언어 모델(LLM)
멀티모달 대규모 언어 모델(MLLM)
시각 언어 모델(VLM)과 시각 언어 행동(VLA) 모델
세계 모델
제3장 AI 기반 모델의 공통 기술
기반 모델의 아키텍처, 관련 알고리즘
시각 처리 알고리즘
트레이닝, 미세 조정 기술
강화 학습
지식 그래프
추론 기술
스파스화
생성 기술
제4장 AI 기반 모델 기업
OpenAI
Google
Meta
Anthropic
Mistral AI
Amazon
Stability AI
xAI
Abu Dhabi Technology Innovation Institute
SenseTime
Alibaba Cloud
Baidu AI Cloud
Tencent Cloud
ByteDance & Volcano Engine
Huawei
Zhipu AI
Flytek
DeepSeek
제5장 자동차에서의 AI 기반 모델 적용 사례
콕핏 케이스
지능형 주행 사례
제6장 AI 기반 모델의 응용 동향
데이터
알고리즘
컴퓨팅 파워
엔지니어링
KTH
영문 목차
영문목차
Research on AI foundation models and automotive applications: reasoning, cost reduction, and explainability
Reasoning capabilities drive up the performance of foundation models.
Since the second half of 2024, foundation model companies inside and outside China have launched their reasoning models, and enhanced the ability of foundation models to handle complex tasks and make decisions independently by using reasoning frameworks like Chain-of-Thought (CoT).
The intensive releases of reasoning models aim to enhance the ability of foundation models to handle complex scenarios and lay the foundation for Agent application. In the automotive industry, improved reasoning capabilities of foundation models can address sore points in AI applications, for example, enhancing the intent recognition of cockpit assistants in complex semantics and improving the accuracy of spatiotemporal prediction in autonomous driving planning and decision.
In 2024, reasoning technologies of mainstream foundation models introduced in vehicles primarily revolved around CoT and its variants (e.g., Tree-of-Thought (ToT), Graph-of-Thought (GoT), Forest-of-Thought (FoT)), and combined with generative models (e.g., diffusion models), knowledge graphs, causal reasoning models, cumulative reasoning, and multimodal reasoning chains in different scenarios.
For example, the Modularized Thinking Language Model (MeTHanol) proposed by Geely allows foundation models to synthesize human thoughts to supervise the hidden layers of LLMs, and generates human-like thinking behaviors, enhances the thinking and reasoning capabilities of large language models, and improves explainability, by adapting to daily conversations and personalized prompts.
In 2025, the focus of reasoning technology will shift to multimodal reasoning. Common training technologies include instruction fine-tuning, multimodal context learning, and multimodal CoT (M-CoT), and are often enabled by combining multimodal fusion alignment and LLM reasoning technologies.
Explainability bridges trust between AI and users.
Before users experience the "usefulness" of AI, they need to trust it. In 2025, the explainability of AI systems therefore becomes a key factor in increasing the user base of automotive AI. This challenge can be addressed by demonstrating long CoT.
The explainability of AI systems can be achieved at three levels: data explainability, model explainability, and post-hoc explainability.
In Li Auto's case, its L3 autonomous driving uses "AI reasoning visualization technology" to intuitively present the thinking process of end-to-end + VLM models, covering the entire process from physical world perception input to driving decision outputted by the foundation model, enhancing users' trust in intelligent driving systems.
In Li Auto's "AI reasoning visualization technology":
Attention system displays traffic and environmental information perceived by the vehicle, evaluates the behavior of traffic participants in real-time video streams and uses heatmaps to display evaluated objects.
End-to-end (E2E) model displays the thinking process behind driving trajectory output. The model thinks about different driving trajectories, presents 10 candidate output results, and finally adopts the most likely output result as the driving path.
Vision language model (VLM) displays its perception, reasoning, and decision-making processes through dialogue.
Various reasoning models' dialogue interfaces also employ a long CoT to break down the reasoning process as well. Examples include DeepSeek R1 which during conversations with users, first presents the decision at each node through a CoT and then provides explanations in natural language.
Additionally, most reasoning models, including Zhipu's GLM-Zero-Preview, Alibaba's QwQ-32B-Preview, and Skywork 4.0 o1, support demonstration of the long CoT reasoning process.
DeepSeek lowers the barrier to introduction of foundation models in vehicles, enabling both performance improvement and cost reduction.
Does the improvement in reasoning capabilities and overall performance mean higher costs? Not necessarily, as seen with DeepSeek's popularity. In early 2025, OEMs have started connecting to DeepSeek, primarily to enhance the comprehensive capabilities of vehicle foundation models as seen in specific applications.
In fact, before DeepSeek models were launched, OEMs had already been developing and iterating their automotive AI foundation models. In the case of cockpit assistant, some of them had completed the initial construction of cockpit assistant solutions, and connected to cloud foundation model suppliers for trial operation or initially determined suppliers, including cloud service providers like Alibaba Cloud, Tencent Cloud, and Zhipu. They connected to DeepSeek in early 2025, valuing the following:
Strong reasoning performance: for example, the R1 reasoning model is comparable to OpenAI o1, and even excels in mathematical logic.
Lower costs: maintain performance while keeping training and reasoning costs at low levels in the industry.
By connecting to DeepSeek, OEMs can really reduce the costs of hardware procurement, model training, and maintenance, and also maintain performance, when deploying intelligent driving and cockpit assistants:
Low computing overhead technologies facilitate high-level autonomous driving and technological equality, which means high performance models can be deployed on low-compute automotive chips (e.g., edge computing unit), reducing reliance on expensive GPUs. Combined with DualPipe algorithm and FP8 mixed precision training, these technologies optimize computing power utilization, allowing mid- and low-end vehicles to deploy high-level cockpit and autonomous driving features, accelerating the popularization of intelligent cockpits.
Enhance real-time performance. In driving environments, autonomous driving systems need to process large amounts of sensor data in real time, and cockpit assistants need to respond quickly to user commands, while vehicle computing resources are limited. With lower computing overhead, DeepSeek enables faster processing of sensor data, more efficient use of computing power of intelligent driving chips (DeepSeek realizes 90% utilization of NVIDIA A100 chips during server-side training), and lower latency (e.g., on the Qualcomm 8650 platform, with computing power of 100TOPS, DeepSeek reduces the inference response time from 20 milliseconds to 9-10 milliseconds). In intelligent driving systems, it can ensure that driving decisions are timely and accurate, improving driving safety and user experience. In cockpit systems, it helps cockpit assistants to quickly respond to user voice commands, achieving smooth human-computer interaction.
Table of Contents
Definitions
1 Overview of AI Foundation Models
1.1 Introduction to AI Models
Definition and Features of AI Models
Classification of AI Models by Architecture
Classification of AI Models by Task Type/Training Method
Classification of AI Models by Supervision Mode
Classification of AI Models by Modality
Application Process of AI Models
1.2 Introduction to Foundation Models
Classification of Foundation Models
Current Development of Foundation Models in Automotive Industry
Application Scenarios of Foundation Models in Automotive Industry
Application Case 1: Application of LLM in Autonomous Driving
Application Case 2: Application of VFM in Autonomous Driving
Application Case 3: Application of MFM in Autonomous Driving
2 Analysis of AI Foundation Models of Differing Types
2.1 Large Language Models (LLM)
Development History of LLM
Key Capabilities of LLM
Cases of Integration with Other Models
2.2 Multimodal Large Language Models (MLLM)
Development and Overview of Large Multimodal Models
Large Multimodal Models VS. Large Single-modal Models (1)
Large Multimodal Models VS. Large Single-modal Models (2)
Technology Panorama of Large Multimodal Models
Multimodal Information Representation
Multimodal Large Language Models (MLLM)
Architecture and Core Components of MLLM
Status Quo of MLLM
Dataset Evaluation by Different MLLM Representatives
Reasoning Capabilities of MLLM
Synergy between MLLM and Agent
Application Case 1: Application of MLLM in VQA
Application Case 2: Application of MLLM in Autonomous Driving
2.3 Vision-Language Models (VLM) and Vision-Language-Action (VLA) Models
Development History of VLM
Application of VLM
Architecture of VLM
Evolution of VLM in Intelligent Driving
Application Scenarios of VLM: End-to-end Autonomous Driving
Application Scenarios of VLM: Combination with Gaussian Framework
VLM->VLA
VLA Models
Principles of VLA
Classification of VLA Models
Application Cases of VLA (1)
Application Cases of VLA (2)
Application Cases of VLA (3)
Application Cases of VLA (4)
Case 1: Core Functions of End-to-End Multimodal Model for Autonomous Driving (EMMA)
Case 2: World Model Construction
Case 3: Improve Vision-Language Navigation Capabilities
Case 4: VLA Generalization Enhancement
Case 5: Computing Overhead of VLA
2.4 World Models
Key Definitions of World Models and Application Development
Basic Architecture of World Models
Framework Setup and Implementation Challenges of World Models
Video Generation Methods Based on Transformer and Diffusion Models
Technical Principle and Path of WorldDreamer
World Models and End-to-end Intelligent Driving
World Models and End-to-end Intelligent Driving: Data Generation
Case 1: Tesla World Model
Case 2: NVIDIA
Case 3: InfinityDrive
Case 4: Worlds Labs Spatial Intelligence
Case 5: NIO
Case 6: 1X's "World Model"
3 Common Technologies in AI Foundation Models
Common Foundation Model Algorithms and Architectures
Comparison of Features and Application Scenarios between Foundation Model Algorithms
3.1 Foundation Model Architectures and Related Algorithms
Transformer: Architecture and Features
Transformer: Algorithm Mechanisms
Transformer: Multi-head Attention Mechanisms and Their Variants
KAN: Potential to Replace MLP
KAN: Cases of Integration with Transformer Architecture
MAMBA: Introduction
MAMBA: Architectural Foundations
MAMBA: Latest Developments
MAMBA: Application Scenarios
MAMBA: Cases of Integration with Transformer Architecture
Applicability of CNN in the Era of Foundation Models
Applicability of RNN Variants in the Era of Foundation Models
3.2 Visual Processing Algorithms
Common Vision Algorithms
ViT
CLIP Scenarios and Features
CLIP Workflow
LLaVA Model
3.3 Training and Fine-Tuning Technologies
Foundation Model Training Process
Training Case: Geely's CPT Enhancement Solution
Instruction Fine-tuning
Training Case: Geely's Fine-tuning Framework for Multi-round Dialogues
3.4 Reinforcement Learning
Introduction to Reinforcement Learning
Reinforcement Learning Process
Comparison between Some Reinforcement Learning Technology Routes
Cases of Reinforcement Learning (1)-(3)
3.5 Knowledge Graphs
Optimization Directions for Retrieval-Augmented Generation (RAG)
Evolution Directions of RAG (1): KAG
Evolution Directions of RAG (2): CAG
Evolution Directions of RAG (3): GraphRAG
RAG Application Case 1:
RAG Application Case 2:
RAG Application Case 3: Li Auto
RAG Application Case 4: Geely
Comparison between RAG Routes
Function Call
3.6 Reasoning Technologies
Reasoning Process of Transformer Models
Evaluation of Reasoning Capabilities
Three Optimization Directions for Foundation Model Reasoning
Reasoning Task Types (1)
Reasoning Task Types (2)
Reasoning Task Types (3)
Common Reasoning Algorithm 1: CoT
Common Reasoning Algorithm 2: GoT/ToT
Comparison between Common Reasoning Algorithms
Common Reasoning Algorithm 3: PagedAttention
Reasoning Case 1: Geely
Reasoning Case 2: NVIDIA
3.7 Sparsification
Characteristics of MoE Architecture
Principles of MoE Architecture
MoE Training Strategies
Advantages and Challenges of MoE
MoE Models from Different Foundation Model Companies
Evolution Direction of MoE
3.8 Generation Technologies
Introduction to Generative Models
Comparison between Generation Technologies
Case 1: Li Auto
Case 2: XPeng
Case 3: SAIC
4 AI Foundation Model Companies
Development History of Mainstream Foundation Models
Mainstream Foundation Models and Their Companies (Foreign)
Mainstream Foundation Models and Their Companies (Chinese)
Rankings of Evaluated Foundation Models
4.1 OpenAI
Product Layout
Product Iteration History
GPT Series: Features
GPT Series: Architecture
From GPT-4V to 4o
Reasoning Model OpenAI o1
SORA: Features
SORA: Performance Evaluation
SORA: Advantages and Limitations
4.2 Google
Development History of Foundation Models
Typical Model BERT: Architecture
Typical Model BERT: Variants
Gemini Model
Cases of Foundation Models in the Automotive Industry
4.3 Meta
LLAMA3.3
LLAMA Series: Evolution
LLAMA Series: Features
LLAMA Series: Training Methods
LLAMA Series: Alpaca
LLAMA Series: Vicuna
4.4 Anthropic
Claude Performance Evaluation
Claude-based PC-side Agent
4.5 Mistral AI
Expert Model: Architecture
Expert Model: Algorithm Features (1)
Expert Model: Algorithm Features (2)
Large Language Model: Mistral Large 2
4.6 Amazon
Nova Product System
Application Cases of Amazon AI Cloud in the Automotive Industry (1)-(3)
4.7 Stability AI
Product System
Stable Diffusion Architecture Based on Diffusion Models
Comparison between Stable Diffusion Video Generation Technology with Competitors
4.8 xAI
Product System
Capabilities of xAI Models
Capabilities of Grok-2
Capabilities of Grok-0/1
4.9 Abu Dhabi Technology Innovation Institute
Iteration History of Falcon Model Series
Parameters of Falcon 3 Series
Evaluation of Falcon 3 Series
4.10 SenseTime
Major Foundation Model Product Systems
Major Foundation Model Product Systems
Foundation Model Training Facilities
Functional Scenarios of Foundation Models
Foundation Model Technologies
4.11 Alibaba Cloud
Foundation Model Product System
End-cloud Integration Solutions of Foundation Models
4.12 Baidu AI Cloud
Foundation Model Product System
4.13 Tencent Cloud
Foundation Model Product System
Reasoning Service Solutions (1)-(3)
Generation Scenario Solutions for Foundation Models
Q&A Scenario Solutions for Foundation Models
4.14 ByteDance & Volcano Engine
Doubao Model System
Functional Highlights of Volcano Engine's Cockpit
4.15 Huawei
Pangu Model Product System
Application Cases of Pangu Models in Data Synthesis
LLM Architecture of Pangu Models
Capabilities of Pangu Models: Multimodal Technology
Capabilities of Pangu Models: Thinking & Reasoning Technology
AI Cloud Services of Pangu Models
4.16 Zhipu AI
Product System
Foundation Model Base in the Automotive Industry
Technical Features
4.17 Flytek
Product System
Functional and Technical Highlights
Cockpit AI System
4.18 DeepSeek
Product System
Technical Inspiration from DeepSeek V3
Technical Highlights of DeepSeek R1
Application Cases of DeepSeek (1)-(3)
5 Application Cases of AI Foundation Models in Automotive
5.1 Cockpit Cases
Lenovo's AI Vehicle Computing Framework Used in Cockpits
In-cabin Functions of Thundersoft's Rubik Foundation Model
LLM Empowers Smart Eye's DMS/OMS Assistance System
Application of DIT in Voice Processing Scenarios
Application of Unisound's Shanhai Model in Cockpits
Phoenix Auto Intelligence's Cockpit Smart Brain
5.2 Intelligent Driving Cases
Li Auto: Multimodal Technology in Autonomous Driving (1)
Li Auto: Multimodal Technology in Autonomous Driving (2)
Li Auto: Multimodal Technology in Autonomous Driving (3): Overcoming 2D Limitations
Li Auto: Data Generation Technology (1)
Li Auto: Data Generation Technology (2)
Li Auto: CoT Technology in DriveVLM
Li Auto: Application of Visual Processing
Li Auto: Data Selection
Geely: Application of Visual Processing
Geely: Multimodal Learning Framework
Waymo: Generative World Model GAIA-1
Tesla: Algorithm Architecture (Including NeRF)
Tesla: Skeleton, Neck, and Head of Vision Algorithms
