For related articles shared by me on network automation, please refer to the catalog "NetDevOps from Scratch"
In recent years, with the continuous development of the global cloud computing field and the continuous growth of business, network technology has also continued to develop, and SDN technology has emerged. From the original core idea of separation of forwarding and control based on Openflow, people continue to expand In the extension of SDN, people can currently reach a consensus that Openflow is no longer a necessary condition (but the separation of forwarding and control is still a core condition), and network programmability has slowly become one of the important criteria for measuring an SDN architecture.
Programmable operations of traditional network equipment are generally based on CLI and SNMP protocols. Whether scripts or network management software, they are all developed on this basis to achieve the broad range of network programmability we are going to talk about today. capabilities, thereby realizing the automation of many scenarios. Some devices support the configuration of some web interfaces and the replacement of the overall configuration through xml. These are very rare and will not be described in detail in this article.
CLI
CLI (Command-line Interface) realizes human-computer interaction through the command line. It is a necessary skill for network workers. People open the software SSH or Telnet to the device every day, then paste a configuration, save it and take effect. One day, people got tired of this kind of repetition, and used a program to automatically generate configuration scripts, log in to the device in batches, and issue configurations to take effect, realizing automation. This is a network programmable method. Let's talk about the advantages, which are very consistent with people's thinking, ideas and existing technical systems. But ultimately, this approach favors people over network devices. It has the following disadvantages:
-There are huge differences in command sets between manufacturers. Not only manufacturers, but different software versions of the same model may have very different differences.
-Developers must be familiar with the command set and how to use it. There are security risks at the configuration level. For example, with a flick of the hand, the port I wanted to open turned into closing the port…
-There are no mandatory requirements for transmission protocols (SSH and Telnet), and there are production security risks.
-The process of parsing and generating configurations is extremely complicated. In many cases, the regular rules written can only be infinitely close to the "truth", but not the whole "truth".
-There is no transactionality, and a configuration may partially take effect and part not take effect.
-There is no automated inspection mechanism and it is completely dependent on people. For example, I want to test whether the generated script is correct. There is a way, but it is very difficult and often difficult to implement easily.
-No idea of data modeling
CLI is always a way of human-computer interaction. It can give the network certain programmability capabilities through programs, but after all, it is not a method that is inherently network programmable. Under the current wave of cloud computing and SDN, it is not suitable for large-scale automated deployment in the network, and its programmability is limited. It is difficult for outsiders to understand the difficulty of development.
SNMP
SNMP (SNMP, Simple Network Management Protocol), this protocol can support network management systems to monitor whether the devices connected to the network have any situation that causes management attention. It consists of a set of network management standards, including an application layer protocol, database schema, and a set of data objects.
For a piece of content in Wikipedia, we highlight network management, monitoring, and data objects. It is used to manage the network, can be configured and collected, and is mainly used for monitoring. It has data modeling to structure some modules, characteristics, and status data of network equipment. It is mainly used for network management systems (mostly monitoring). Then let's talk about its shortcomings:
-Poor readability. It prefers the "machine" in human-machine. It is not readable when used, and the modeling data is also not readable. It uses a superset of ASN.1.
-The security is limited. There are three versions: v1, v2c and v3, and the security is improved in sequence. However, the most common one is v2c, which has limited security. The v3 version is very safe by design, but it is not universal. . .
-There is no backup, recovery or rollback mechanism. We also have show run and other methods to back up the command line, but snmp. . .
-Very few writes. Read a lot, write little, mostly used for monitoring.
-The data items that can be collected are limited, and the configuration of the entire device cannot be obtained. Many times we find that we can use cli to collect it, but we cannot use snmp to collect it.
-There is a performance bottleneck. The upper limit of collected data is 64K, and the collection granularity is too large. In large and complex networks, it may take minutes or longer. This also highlights the important point. Our requirements for granularity are also very strict. Many times we hope to collect port traffic every few seconds. In large networks, I think traditional network management software is… To expand on one more sentence, the current method is Telemetry (such as gRPC) which can achieve microsecond level, and some require a combination of software and hardware. It is not yet popular, but in the future It must be a trend. As for when this will come in the future…
-Since its birth, SNMP has been greatly used in the field of network monitoring to obtain data for monitoring. The lack and complexity of configuration capabilities have led to little use of it in network configuration. Read-only network programmable.
Netconf Protocol and YANG Model
Facing the next generation of networks, what kind of network management protocols do we need to better realize network programmability and improve the level of automation?
The IETF proposed the following ideas in RFC3535 in 2002 (actually there are 33 of them. Based on online information and the author's knowledge, I wrote the following ideas):
1. There is a programmable interface for network configuration
2. The same configuration can be used across manufacturers and models
3. Need to unify a modeling language with good readability
4. Complete error checking and recovery functions
5. Transactional
If you have an idea, just implement it. In 2006, the IETF proposed the Netconf protocol, which solved the problems raised by RFC3535. The initial Netconf only stipulated the basic framework and operations of the protocol, and defined solutions that took into account some problems of RFC3535. It did not stipulate a unified modeling language. Therefore, some early manufacturers' equipment only supported some basic operations of Netconf, and did not use a unified bottom layer. Data modeling language.
RFC6020 was released in 2010, proposing the YANG Model modeling language and a method of combining it with NETCONF. One definition is a data modeling language that unifies the underlying resource logic between manufacturers, and the other definition is a unified command set for each manufacturer's operations on configuration data and status data. The data instances created by the YANG model are wrapped in the Netconf protocol. Transmission, the two are combined with each other to build a new set of universal network programmable interfaces for the new era based on the YANG model and driven by the Netconf protocol.
After 2016, the Netconf protocol was closely integrated with the YANG Model and became popular. So far, when we look at some SDN architecture software aspects, we have heard these two terms more or less.
YANG and Netconf, one is static and the other is dynamic, just like yin and yang. The two have derived the network programmable world of the next era. (When we look at the YANG warehouse on github, we will also find that its icon is Tai Chi, and the connection between its name and "Yang" somewhat reveals the original designer's design ideas).
Next, we will briefly talk about the YANG Model and Netconf protocol. Let's first talk about the data modeling language YANG to see how it describes the digital twin of this network world.
YANG Model
In the RFC6020 document, the opening chapter clearly states, YANG, A Data Modeling Language for the Network Configuration Protocol. It is the abbreviation of Yet Another Next Generation (Yang) Data Modeling Language. It is a modeling language used to describe network concepts.
Supports definition of lists, dictionaries, and even more complex data structures, supports constraints, enumerations, reference imports, version management, and namespaces. Due to space, we will give a brief explanation. For detailed information, you can refer to:
It can describe this network device very simply in a structured language. For example, for the definition of a port:
As a professional operation and maintenance personnel, with a little bit of network basics and a little bit of programming basics, you can understand the definition of a port relatively clearly. It is a list structure, and there can be multiple. One of its attributes is interface-name (also a key). , unique, non-repeatable), as well as the speed attribute and duplex attribute, both of which are strings.
Many attributes of a network device are described by the YANG Model, including configuration status and operating status.
In this way, YANG Model describes the online world using structured language. If you are interested, you can read the above Internet blog post, which has a very in-depth description.
It can be converted into XML data very well and wrapped in the Netconf protocol for transmission (we will explain it later):
At the same time, in order to level the differences between vendors, Openconfig, led by Google, has standardized the data model. From the official website, we see the slogan "Vendor-neutral, model-driven network management designed by users", which is designed by users and cross-platform. Vendor-common, model-driven network programming (let's translate it this way first). To put it simply, it is to make the modeling between various manufacturers the same, so that when you configure certain data, you don't have to look through each manufacturer's private yang model one by one. But the Internet always has private protocols, and different manufacturers will always create new and better private protocols for "better user experience" and "better business strategy" (this is really the original sin of network manufacturers). The picture shows some of the more commonly used openconfig yang model implementations.
Judging from the picture, I think there are quite a lot of them, and the commonly used configurations are relatively complete. But in practice, it depends on whether the manufacturer also supports these yang models. Some higher version devices of a certain subject are basically supported. I haven't taken a closer look at the domestic ones yet.
Networks cannot be exactly the same. For an engineer who is engaged in network operation and maintenance development, it is a blessing to be able to achieve the same goal!
openconfig can be found in https://github.com/openconfig/public/tree/master/release/models
You can find private yang models on various official websites.
Netconf protocol
After talking about the yang model, let's talk about the Netconf protocol. The yang model defines the digital description of the network world, and Netconf defines the acquisition (get) and adjustment (config) of data.
Netconf encapsulates the data of the world described by the yang model to realize the management of the network world.
Yang data is encapsulated in xml and then managed through the Netconf protocol. It is a protocol with a great layered idea, describing some details of the protocol in a hierarchical manner. Let's look at the picture above.
-Transmission: Netconf is transmitted through the SSH protocol, is connection-oriented, and has security guarantees.
-Message: Make a remote call to the network device through RPC, the network manager issues an rpc request, and the network device resumes rpc-reply.
-Operation: This is the soul of Netconf. It supports get (configuration and running data), get-config (getting configuration data, and a device can have multiple configuration data, one running, one startup, multiple candidate candidates), edit -config (configure network device parameters, supports addition, deletion and modification), delete-config, copy-config (copy the configuration to the destination, the destination can be ftp, file or running configuration, etc.), lock\unlock (lock the configuration to prevent configuration conflicts or failures caused by multi-process operations) and so on.
-Data: data is yang data wrapped in xml. Like the port we described above, structured data is easy to program. Used to describe the data to be configured or deleted or obtained.
These are the four layers of Netconf. The control end and the network device communicate through Netconf, through the traditional ssh protocol, using the Netconf subsystem, and the default port is 830. As shown below:
This figure demonstrates the interaction using raw ssh, but in fact we implement this process through programming. I will demonstrate the programming implementation method to you later.
Netconf configures network devices. The interaction process is roughly as follows:
This picture is so low, you can also see that it was drawn by me… My understanding of Netconf is as above. I think there are many pictures on the Internet that are not right, and many behaviors of the server agent are not correct. This is what I intuitively feel when I log in to the device, and of course it corresponds one-to-one with the official documentation.
We can look at some Netconf examples:
Hello, build a link.
We saw several keywords, Netconf version, supported YANG Model, session id. At the same time, hello indicates what namespace we are operating in. In this case, it is the corresponding version of Netconf.
Get configuration
One parameter of get-cofig is source, which is where the configuration data is obtained (running, startup or other). Another parameter is filter, that is, which data is obtained from the data model described by which yang model. This corresponds to the capability originally sent by the network device. If successful, the corresponding configuration data will be returned.
Get configuration or running data
Similar to get-config, but what is obtained is running configuration (personal understanding) or running data. Filter can be specified.
Copy configuration
The copy operation has two parameters, source and destination. The successful reply is with the ok tag.
Edit configuration
When editing the configuration, specify the data item to be edited, the namespace of the capability and the corresponding label. For example, this is to configure dhcp, which is described by the yang model http://tail-f.com/ns/example/dhcp.
Close session gracefully
It is this kind of message that is transmitted back and forth in ssh. We just extract part of the message to facilitate everyone's understanding.
Then simply add some content for reference.
-Netconf is based on session, and every success will have a session id.
-Each request has a message id, as long as it gradually becomes larger
-Data configuration can be locked, exclusive, and operated through lock.
-Netconf is transactional, and operations are either all implemented or none. At the same time, according to the official website documentation, this transactionality is for the configuration of N network devices, that is, one-time configuration polymorphism can support transactionality. But I haven't done it yet…
-Netconf supports subscription. In terms of device performance, the order of magnitude is about 5 sessions. I can subscribe to a certain data item and the device will notify me when it changes.
-Capability, this is how I understand it. The network device sends the version of Netconf and YANG Model, and the control terminal sends the version of Netconf. Only when the Netconf version matches the two can we continue. This is my intuitive feeling. Any advice is welcome.
-Operations such as get edit will specify the data to be changed, which can be filtered using filter.
-copy-config supports copying a complete set of configurations from somewhere to somewhere. The somewhere can be an FTP File, running, startup, and candidate configurations on the device.
-Netconf also supports verification of configuration, using the validate operation.
This article still hopes to popularize science, and I will not go into details. You can read the relevant protocols of RFC, which is actually not very long.
In practice, based on some open source software, such as python's ncclient, we can easily configure network devices automatically and achieve network programmability. This is the mission of Netconf and YANG Model.
Network personnel read the well-formatted YANG Model definitions and use relevant programming languages to perform programmable operations on network devices based on the operations defined by Netconf. In this way, the path to network programmability is forged.
Let's expand and imagine that the YANG Model has defined the data structure of the network device. We can operate it through Netconf. Can it also be operated through other protocols?
The answer is yes. In fact, many other protocols have been derived from Netconf, such as RESTConf. As shown below,
YANG Model (public and native) defines the data structure, above which are new network management protocols, Netconf, RESTCon, gRPC, etc. In this way, we can operate network devices through RESTConf based on HTTP RESTful API, we can also operate network devices through Netconf based on SSH, or we can operate network devices through gRPC based on HTTP2.0. They are all based on YANG with good data structure. Model, write the corresponding data, encapsulate it into xml or json to program network devices. This is the future of network programmability. To be precise, it is Model Driven Program, model-based network programmability. Network engineers gradually focus on the parameters of the device instead of the command set, and configure the network parameters by reading the corresponding data model.
At the end, I write, why should I open this public account. I studied computer science and technology when I was in school. After entering the workplace, I was engaged in network operation and maintenance work. Thinking about it, the reason why I was divided into teams may be because I was a graduate student at the Network Technology Research Institute (manual funny). From the very beginning, I was involved in network operations. In the later stage of operation and maintenance, tools were used to simplify work and improve efficiency based on CLI. Later, the tools were gradually developed into BS-structured web applications. They were constantly exposed to new technologies and continued to enrich new functions.
Fortunately, they caught up with the development of open source technology and SDN, and gradually I transitioned into NetDevOps work and used my programming skills to improve the team's operation and maintenance capabilities. I also enjoyed writing this line of code. As the writing progresses, it is gradually discovered that NetDevOps should be a skill that every network engineer should have in the future (everyone adds fuel to the fire), so that they can achieve both high-level planning and rapid implementation. Looking back at some information on the Internet, to be honest, there is very little in China, and the domestic atmosphere is not very strong. Many domestic software are based on the old CLI and snmp, and everyone still uses text tools and SSH tools for work. So I hope that I can teach others how to fish, share my experience (pits) and skills with more network operation and maintenance engineers , and do my best. Xiao Chu said that you can learn something to reduce your workload, and by focusing on the distant future, domestic network operation and maintenance can truly evolve towards automation.
In the future, I will record some videos and write some articles. It feels really strenuous to write a document. You are welcome to subscribe, collect, click like and watch.
appendix: Netconf common operations
DWDM OTN Solution Design and Cost Quotation, pls link with me, Taylor Huang
