In this blog, I am going to tell you about the MPTCP Proxy support for ATSSS in Truminds 5G UPF. So, first of all let’s understand some key terms quickly.

UPF stands for the User Plane Function. It is a 5G network function which routes traffic from mobile side to internet and vice versa. You can imagine that a UPF must handle packets from tens and thousands of mobiles if not more, so it is operating at scale and must be highly efficient in its processing.

ATSSS stands for Access side Traffic Steering, Switching Splitting. So, the term Access simply means the interface of the UPF towards the mobile side. I always imagine in my head that the mobile is towards the far left, UPF is in the middle and the Internet is on the right side. Something like this –

The interesting part begins from the fact that the Mobile really has two main data interfaces. One is the Cellular interface and the other is the Wifi interface. One can hardly imagine a mobile which will not have a Wifi interface these days. Let us also keep in mind that the Mobile is trying to reach some server on the Internet. Let us call that server as ServerOnInternet. Let us further assume that the connection between the Mobile and the ServerOnInternet is a TCP connection. So, let us imagine the below picture.

In normal scenarios, the TCP connection from Mobile would use either the Cellular interface or the Wifi interface. Afterall, it is one TCP connection, and one TCP connection uses one source IP address and therefore one interface on Mobile which is having that IP address. The UPF merely shunts the IP packets right and left, but the TCP connection is end-to-end between the Mobile and the ServerOnInternet

Now the above is a pity, because even though we have an additional Wifi interface on the Mobile, for the above TCP connection we are ending up using just one interface – the Cellular one in our usecase. If TCP allowed it, it would have been nice to somehow use the additional Wifi interface too to send data from there too thus fattening the pipe of data between Mobile and ServerOnInternet. Alas, TCP would not allow us to take advantage of this.

Now, while TCP does not allow to use multiple interfaces for a TCP connection, there is a protocol called MPTCP which does. That is why there is the additional MP in the MPTCP. MP stands for Multipath. As the name indicates, MPTCP allows a client machine to make multiple TCP connections towards a server which supports MPTCP. For application on the client, it is still a single connection like the TCP connection earlier. But under the hood, this MPTCP connection comprises of potentially multiple individual TCP connections between the client and the server. Since each TCP connection can use its own interface and IP address, so in simple terms, in our usecase, the mobile can make an MPTCP connection with ServerOnInternet comprising of two TCP connections – the first TCP connection will use the Cellular interface and the second TCP connection will use the Wifi interface. And voila, we have solved the problem, haven’t we ! And we did not have to do anything special at the UPF either. MPTCP clearly rocks !

But there is always a catch. And this will be no exception. The trouble is that while the latest Mobile devices meant for 5G are sure to support MPTCP, the various servers on the internet may not support MPTCP just yet. What if the server on the internet does not support MPTCP, but just supports TCP ? Well, in that case when the Mobile attempts to make an MPTCP connection to the ServerOnInternet which does not support MPTCP, the MPTCP specification has defined procedures to let the Mobile client fallback to normal TCP after detecting that the server does not support MPTCP. So we are back to square one. If most of the servers on the Internet do not support MPTCP (unfortunately that is the sad truth for the moment), then the Mobile would only be able to use just one interface for the connection to the server.

The clever 3GPP folks anticipated the above issue and came out with a solution. They said, ok we accept the fact that the servers on the internet may not support MPTCP, but the Mobile would indeed support MPTCP in 5G. Think for a moment, what solution they came up with. You can take a small pause from reading further and think how they did it. Hint, they used the UPF for this.

Alright, so what the 3GPP folks said was that the UPF would be required to support MPTCP in 5G as a feature. So now we have a situation where the Mobile supports MPTCP, the UPF supports MPTCP but the ServerOnInternet does not support MPTCP in our example usecase. So, what the Mobile would do is that it will make an MPTCP connection from itself to the UPF. Essentially the mobile would make a TCP connection over its Cellular interface and another TCP connection over the Wifi interface. These TCP connections comprising the single MPTCP connection would terminate at the UPF which also supports MPTCP. But remember that the goal of the mobile is to exchange data with the ServerOnInternet and not just with the UPF. So, the UPF makes a regular TCP connection from itself to ServerOnInternet. Now the data from Mobile comes on MPTCP to the UPF, the UPF then sends this data on TCP to ServerOnInternet. The ServerOnInternet sends data on TCP to the UPF, the UPF sends this data on MPTCP to the Mobile. This is what the MPTCP proxy on UPF does. Pretty neat ! So something like this –

There is still a fly in the ointment though. How does the Mobile tell the UPF what is the IP address of the ServerOnInternet it is trying to reach out to ? Recall that the Mobile is just making a bunch of TCP connections to the UPF IP Address as part of its MPTCP connection. That is where the solution in RFC 8803 comes into play. The long and short is that when the Mobile sends a TCP SYN for the first TCP connection it makes to the UPF, it sends the information about the details of the IP Address and port number it intends to reach out to for the ServerOnInternet. So now the UPF knows enough information to make the TCP connection to ServerOnInternet and then proxy the data back and forth like I explained earlier. Game set and match. Yes indeed ! the ladder diagram below from the 3GPP TS 29.244 gives a nice little overview of what happens under the hood

Now then, we must think of some practical issues in UPF for handling all these MPTCP and TCP connections at scale. As I mentioned earlier, there will be tens of thousands of mobiles trying to make connections to various servers on the internet. This means there will be tens and thousands of MPTCP and TCP connections to be handled at UPF. A typical UPF, as is the case with Truminds UPF, is implemented in user space with high speed I/O enabled by DPDK (Data Plane Development kit) to bypass the Linux kernel. If we do bypass the Linux kernel, then we do not have the opportunity to use the Linux TCP or MPTCP stack. We are pretty much on our own. So, the Truminds engineers recognized this and wrote a user space TCP and MPTCP stack of their own which runs in context of the worker threads of the UPF which run on the various CPU cores of the machine where UPF is running. Implementing a TCP stack and then MPTCP on top of it is a daunting task, but Truminds engineers did it. The stack itself is generic so if you are building your own UPF and want a similar usecase to be achieved, you could either write your own TCP and MPTCP stack or give me a call ! There were other practical challenges on how to use this MPTCP/TCP stack at the application level in UPF which would do the shunting of the packets left and right, the proxying if you will, because this shunting needs to fit in with the erstwhile IP data pipeline of UPF. The interface towards the Access side is a GTP encapsulation, but let us not worry too much about it for the moment, it is possible to overcome those challenges and that makes a topic of a separate blog altogether. The bottom line is Truminds UPF works with this usecase and we are excited about this and can help you too with your similar usecase !

The whole bigger picture below describes an end to end view for those who are deep into the implementation of UPF. The picture has been taken from 3GPP TS 29.244

Before I pen off, here are some important IETF RFC’s which are utilized in the solution –

RFC 793 – (TCP , there are a bunch of other RFC’s, but this is the base RFC for TCP)

RFC 6824, 8684 --  (MPTCP version 0 and version 1)

RFC 8803 (this RFC contains the trick amongst other things so that the Mobile can tell UPF which server on Internet it is trying to reach out to)

3GPP TS 29.244 (this is not an IETF RFC but the 3GPP specification document defining the interface between the Control Plane and UPF, you would find good information there about the MPTCP Proxy in UPF)

Hopefully, you enjoyed reading this blog. Keep tuned for more !

Author: Prashant Upadhyaya and Pankaj Choudhary