Ever wonder how the google language converts an entire webpage into an entirely different language in a matter of seconds and how your phone groups images in your gallery based on their location? All of these are a product of deep learning. Deep learning is a subset of machine learning which in turn is a subset of artificial intelligence. Artificial Intelligence is a technique to enable machines to mimic human behavior, machine learning is a technique to achieve artificial intelligence through algorithms trained with data, and finally, deep learning is a type of machine learning inspired by the structure of human brains. In terms of deep learning, the said structure is called an artificial neural network. Deep learning distinguishes itself from machine learning by the type of data that it works with and the methods in which it learns. While machine learning typically predicts the outcome based on a trained data set, deep learning using artificial neural networks attempts to mimic the behavior of the human brain through a combination of data inputs, weights, and a bias. These elements work together to accurately recognize, classify, and describe objects within the data. Unlike classical machine learning, artificial neural networks also use intermediate layers for data optimization, making it more reliable, efficient, and accurate than classical machine learning in predicting and reorganizing outcomes of data. Having these advantages, deep learning is used in multiple fields of technology including but not limited to computer vision, customer experience, language recognition, and autonomous vehicles.
Experiments on making an autonomous car have been performed since 1920 and some promising trials took place in the 1950s. The first self-sufficient and a fully autonomous car appeared in the 1980s with Carnegie Mellon University's Navlab and ALV projects in 1984, Mercedes-Benz and Bundeswehr University Munich's Eureka Prometheus Project in 1987. With these, major companies such as Mercedes-Benz, General Motors, Continental Automotive Systems, Autoliv Inc, Bosch, Nissan, Toyota, Audi, Volvo, Vislab from the University of Parma, Oxford University, and Google also started investing in autonomous vehicles. With driver-less cars and deep learning implementation in autonomous cars with all other electronic components of advanced technology, it surely provides a soothing feeling to the public, but as every coin has two sides. There are both advantages and disadvantages to these autonomous cars.
Making automatic systems as complex as autonomous vehicles is a real challenge as one has to deal with processing loads of data at a given point in time. But if we have to define where deep learning is used in autonomous vehicles, then there come four basic yet very effective areas where deep learning is applied and helps a vehicle to localize from point A to B.
1. Perception
Perception is used by the system in autonomous vehicles to find the environment and obstacles around the vehicle. It is one of the pillars on which the vehicle relies and contains deep learning.
For perceiving the lanes and obstacles around you, the autonomous vehicle uses three sensors i.e. camera, LiDAR (Light Detection and Ranging), and RADAR (Radio Detection and Ranging). These sensors are used in combination or independently to detect the lanes and surroundings of autonomous vehicles. Algorithms such as LaneNet are used for detecting lanes, and YOLO or SSD is very popular among designers of neural networks in deep learning for autonomous vehicles.
2. Localization
Localization is one of the second pillars and building systems for autonomous vehicles. Localization is all about the location of the vehicle in the world with the highest possible accuracy. Conventional location determination technologies like GPS are not at par with the accuracy of localization methods used in autonomous vehicles and developed via deep learning.
Based on the type of algorithm used for localization, we have mainly 3 methods discussed as follows:
Knowing the Map and Initial location - It is when the vehicle knows the initial location and the destination and has access to the map so it just needs to follow the route to reach the destination.
Knowing the Map but not the initial position - When the vehicle has access to the map but does not know the initial location of the vehicle, then the vehicle uses landmark detection to know the initial position. To do this, the vehicle uses Kalman and Particle Filters.
Knowing neither the Map nor the initial location - It is the condition where the system in the vehicle does not know both the location and the initial position. It is called Simultaneous Localization and Mapping (SLAM) i.e. the vehicle has to know both the map and the location. The SLAM uses Bayesian filtering to obliterate the present problem but nowadays, it uses Visual Odometry to do the same.
3. Planning
Planning in deep learning is the most important aspect as it involves decision-making. It decides what path to be taken to go from point A to point B.
Planning involves three kinds of planning i.e. High Level/Global Planning, Behavioral Planning, and Path/Local Planning. High level/Global Planning involves programming the route from one point to another for which the system uses Graph Search algorithms such as Dijkstra, A*, DFS, and BFS. Behavioral planning involves what other obstacles would do and make decisions based on it and uses Learning-based approaches such as Gaussian Mixture Models for Intent Prediction and Kalman Filters. Local planning involves planning about avoiding obstacles and creating a trajectory using algorithms such as Rapidly-exploring Random Trees (RRT), RRT*, Probabilistic Roadmaps (PRM), and PRM*
4. Control
Control is all about making the vehicle follow the selected path/trajectory to make the vehicle reach from point A to point B. The control is provided by generating steering angles and acceleration values.
Patent Analysis
The development of any technology takes several iterations to be commercialized and used in the public domain. A patent plays a crucial role in identifying and documenting any new invention in any field. Apart from providing royalties to the inventor, it also provides a deep insight into the technology and how it can be used to make a product. Discussing deep learning in autonomous vehicles, nearly 1.86 Lakh patents are present discussing the inventions related to this field. From this number, 51,380 patents are active and granted, 1.39 patent families are alive and ending, and 10,051 are dead.
Top 10 Players
Below are the top 10 players and how many patent families they have related to deep learning in autonomous vehicles. The data of the patent families shown in the graph are active and granted patents.
Tencent Technology Shenzhen is the leading company that has the most number of patent families in deep learning in autonomous vehicles technology with 840 active and granted patent families, followed by IBM with 790, and then LG Electronics with 695 patent families. Those are the top 3 companies followed by Ford, Samsung, Intel, GM, Waymo, Toyota, and TSINGHUA University with data provided in the above graph.
Top 10 Markets
Below are the top 10 markets and how many patent families they have related to deep learning in autonomous vehicles. The data of the patent families shown in the graph are active and granted patents.
China is leading the race in the number of patents in deep learning in the autonomous vehicles field with 28443 patent families. The second is the USA with 21009 patent families, and the third is Korea with 9349. Then follows the European Patent Office with 7727 patent families, followed by Japan with 6811, followed by Germany with 4096 patent families, followed by World Intellectual Property Organization with 3243, followed by Great Britain with 1921, followed by India with 1467 and in last France with 1367 patent families.
Patent Application Trend Over Last 10 years
Below is the trend line of the previous 10 years. The trend line shows the number of patent applications filed in each year comprising the inventions related to deep learning in autonomous vehicles.
According to multiple designers working in the autonomous vehicle industry, the advanced sensing and computer hardware on the vehicles requires a lot of stable electric power which conventional fuel-based engine cars cannot provide. Therefore, the majority of the autonomous vehicles developing are electric vehicles as the commercialized development of electric vehicles are also done in the same era and around the same timeline. Waymo, a popular name in the autonomous vehicles industry, also makes electric-based autonomous vehicles. Although the development of autonomous vehicles started in the 1920s, a perfect commercialized product has yet to be launched. This fact is evident by the maximum number of patent applications filed on deep learning in autonomous vehicles in the late 2010s. The maximum number of patent applications filed were 13068 in 2018 and also 12310 in 2019. The fact that the maximum number of deep learning in autonomous vehicles patents were filed in the late 2010s is also evident by the growing need of producing more electric vehicles due to increasing pollution in the same period. Many environment-related summits and conferences were also conducted in the same period i.e., 2010s. The United Nations Conference on Sustainable Development in 2012, The 7th Digital Earth Summit in 2018, and The Santiago Climate Change Conference are the perfect examples stating the reason for filing a large number of patents in deep learning in the autonomous vehicles field in the late 2010s. Furthermore, the graph shows the expected number of patent applications to be filed in the next year i.e., 2022.
Conclusion
According to an FnF market research report, the market size value of autonomous vehicles was USD 23.33 billion which is expected to grow to USD 64.88 Billion by the year 2026. The expected CAGR growth is 22.7% from the year 2021-2026 with 2020 as a base year. Though major heavyweights including but not limited to Google, BMW, Bosch, and Daimler AG are investing and developing semi-autonomous and fully autonomous vehicles, the stage still is far from where a common man will see the use of autonomous vehicles in their daily routine effectively. Also, the adverse effect of the COVID-19 pandemic on the automotive sector has pushed the development of all kinds of vehicles backward. "Change is inevitable" - with this quote I hope neither the pandemic and neither the shortcomings of autonomous vehicles today will exist in the near future and with the development of technology each day, that day isn't far when we will see only autonomous vehicles on the roads with much larger efficiency and competency that what it is today!
References
https://www.thinkautonomous.ai/blog/?p=deep-learning-in-self-driving-cars
http://www.halffastchicago.com/advantages-and-disadvantages-of-autonomous-car/
https://kmatrix.kaist.ac.kr/urban-mapping-for-autonomous-car-with-slam/
https://spectrum.ieee.org/realtime-robotics-motion-planning-chip-autonomous-cars