pkkm 9 hours ago

You don't need to make up your own for this experiment. There's already a pretty old protocol that's far superior to TCP, but failed to get adoption because of network hardware dropping everything other than TCP and UDP. It's called SCTP.

  • jeroenhd 8 hours ago

    SCTP is fascinating because it's one of the backbone technologies that makes communication possible for most people on the planet (as the mobile network stack pretty much relies on it), yet it's effectively unsupported on almost every consumer device. If you want to use it, you're probably going to have to ship a userland implementation that needs privileges to open a raw network socket, because kernel implementations are rare (and often slow).

    We could've had it as a QUIC replacement if it weren't for terrible middleboxes and IPv4 NAT screwing everyone over once again. Hell, NAT wouldn't even have been an issue had SCTP support been widespread before consumer NAT devices started being implemented.

    It's a prime example of how new network protocols now have to lie and deceive to work over the internet. TLS needs to pretend to be TLS 1.2, QUIC needs to pretend to be an application on top of UDP while reimplementing SCTP and TCP, and even things like secure DNS are now piped over HTTPS just in case a shitty middlebox can't deal with raw TLS.

    • tsimionescu 7 hours ago

      While the gist of your post is spot on, I do feel it should be noted that DoH is preferred over DoT not to protect from middleboxes that don't work properly, but from middleboxes that are actively trying to outright censor encrypted DNS, but can't afford to snoop on/prevent all HTTPS traffic. It's an anti-censorship measure, not a compatibility measure.

    • bewo001 3 hours ago

      Actually, every browser supporting webrtc datachannel supports SCTP over UDP.

    • tepmoc 6 hours ago

      Diameter in mobile network is heavy user of SCTP, but from what I've read they moving away from diameter into HTTP calls for 6G.

      • bennettnate5 4 hours ago

        For communication between carriers and some communications within a given carrier's network components, yes. Base stations all still communicate with the core network over SCTP, though.

    • nesarkvechnep 3 hours ago

      As with most of the network stack, *BSD implementation is the reference implementation.

    • nly 6 hours ago

      It gets worse in practice. Congestional control algorithms on the internet need to play nice with TCP Cubic regardless of how good they are.

    • api 3 hours ago

      It’s too bad the original IP could not have included some kind of stronger header integrity lock to block middle boxes.

      It would have forced us to adopt V6 and… my god… the network would be so much cleaner and superior in every way. Things would just have addresses, and protocol innovation at L3 would be possible.

      • hylaride an hour ago

        I'd like to agree with you, but IPv6 had such a classic case of suffocation by committee during its birth that I doubt it would have been pushed by such a situation. IPSec was mandated in the original IPv6 RFC, for god's sake. That alone delayed a lot of work in implementing it as crypto code needed to be integrated into kernels, which was not common in those days. That's to say nothing about the fact that IPSec is loosely defined enough that setting it up between different vendors is always an adventure - adding support to an IP stack was a big headache (I followed OpenBSD at the time they were integrating IPv6 in the early 2000s and there was a lot of hard problems around compatibility).

        Header integrity was so far off of consideration during IPv4's implementation because the internet was a dozen universities and DoD sites that it was overkill (and possibly a waste of limited CPU cycles at the time).

        What's far more likely to have happened is that we'd see more proxies instead of NATs (SOCKS, etc). I don't think that'd be better than NAT.

        • api an hour ago

          There's some history I'm not aware of here. I didn't know just how bad the V6 second system committee shitshow was.

          Today V6 has been stripped down almost to what it should have been: bigger addresses, then stop. All that's left to get there is to deprecate SLAAC or make it optional.

          • kstrauser an hour ago

            SLAAC’s awesome, though. It’s one of the good parts we definitely want to keep.

            • hylaride 30 minutes ago

              SLAAC is awesome, but DNS support in it didn't show up until much later. The result is a mish-mash of DHCPv6/SLAAC support (and android famously doesn't support DHCPv6 at all and windows only supported DHCPv6 until windows 10).

            • api 35 minutes ago

              Let's say your ISP gives you a /64. Now you have to use V6 NAT... or assign a /96 internally. SLAAC won't let you do that.

              That, among other things, is a problem. SLAAC is too limited.

              You can use DHCPv6 but then you can't use Android because Android, and I think they're alone here, stubbornly and dogmatically refuses to implement it. I guess you could go around and statically assign V6 IPs to Android devices, or run NATv6 with SLAAC for those and DHCPv6 for everything else, but that's annoying as hell.

  • valorzard 8 hours ago

    SCTP is really cool, I first found out about it because it’s the basis for WebRTC data channels. It’s basically reliable UDP, but you can turn off the reliability if you want. Makes me wonder why QUIC exists when SCTP does…

    • 15155 8 hours ago

      > why QUIC exists when SCTP does

      Because QUIC uses UDP, which is supported by most/all intermediate routing equipment.

      • noam_k 8 hours ago

        Is this a real issue? SCTP runs over IP, so unless your talking about firewalls and such, the support should be there.

        Edit: a quick search showed that NAT traversal is an issue (of course!)

        • Karrot_Kream 7 hours ago

          Yes this is called protocol ossification [1] or ossification for short. Other transport layer protocol rollouts have been stymied by ossification such as MPTCP. QUIC specifically went with UDP to prevent ossification yet if you hang out in networking forums you'll still find netops who want to block QUIC if they can.

          [1]: https://en.m.wikipedia.org/wiki/Protocol_ossification

          • crims0n 5 hours ago

            Because from an enterprise security perspective, it breaks a lot of tools. You can’t decrypt, IDS/IPS signatures don’t work, and you lose visibility to what is going on in your network.

        • evrimoztamur 8 hours ago

          Hole punching is perhaps why UDP is de facto?

        • immibis 3 hours ago

          Every home user is behind a NAT. While you can send any protocol between datacenter servers, IPv4 home users are stuck with TCP or UDP.

      • nly 6 hours ago

        The whole point of UDP is to allow alternative protocols to be implemented on top.

        SCTPs mistake was it wasn't implemented as a userland library on top of UDP to begin with.

      • saurik 7 hours ago

        WebRTC runs SCTP over DTLS over UDP.

      • Imustaskforhelp 2 hours ago

        I am now genuinely wondering

        Maybe its me being stupid but why don't we use quic always instead of tcp?

        I think it has to do with something that I read that tcp can do upto 1000 connections simultaneously no worries and they won't interfere with each other's bandwidth / impact each other , but udp does make it possible for one service being very high to impact other.

        There was this latest test by anton putra with udp vs tcp and the difference was IIRC negligible. Someone said that he should probably use udp in kernel mode to essentially get insane performance I amnot sure

    • signa11 4 hours ago

      > It’s basically reliable UDP, ...

      more importantly though, it transmits multiple independent streams of message chunks in parallel.

      similarity with UDP ends at message oriented nature of the protocol. closest equivalent for TCP would be MPTCP I suppose ?

    • bdd8f1df777b 5 hours ago

      Others have mentioned protocol ossification which is indeed the primary reason. A secondary reason is that QUIC fuses TLS so its latency is further reduced by one RTT. For high latency networks, the difference is palpable.

  • nubinetwork 7 hours ago

    I thought the point was that they wanted to use something that didn't exist yet (either in RL use or in RFC form)...

  • immibis 3 hours ago

    It's actually universal within a certain niche. I think phone networks are doing just about everything over SCTP internally. When SS7 links get replaced they get replaced with something that uses SCTP. Not sure of the details because I don't work there.

    Related: there's a parallel Internet with a different root of number allocation called GRX/IPX (GPRS Roaming Exchange/Internetwork Packet Exchange)

Emjayen 4 hours ago

As someone who has implemented various transport protocols (both standard and custom) the biggest hurdle in layering atop IP will not be the gauntlet of WAN routers, surprisingly, rather consumer NAT devices.

One interesting case for a particular Netgear family of routers was that, while the traffic would survive end-to-end it was not without corruption - very specific corruption which took the form of zeroing the first 4 bytes. Given this aligns with where the src/dst ports would normally be, I suspect it was being treated as TCP/UDP albeit without the actual translation path taken.

LinuxBender 3 hours ago

What would happen if we didn't use TCP or UDP?

One would have a hard time communicating with anyone. The internet has standardized around TCP and UDP. There are too many devices in most paths that will not be able to handle other protocols. Replacing all the hardware on the internet would take even longer than deprecating IPv4 in my pessimistic opinion. To get around this there would have to be some significant gains of the new protocol that would warrant big expenses in every corporation and government and all the hardware manufacturers would all have to agree on implementing support and further agree on interpretation of the new RFC.

  • mrbluecoat 2 hours ago

    Agreed. This statement sums up the obvious:

    > Granted, the server was just two hops away from the client, and it didn't have to pass through the scary sea of the internet

jumperabg 11 minutes ago

We would be sending data via rockets guided by pigeons inside of them.

paweladamczuk 4 hours ago

I always assumed any TCP/UDP packets would get captured by the OS network stack in order to be sent only to the processes listening on specific ports.

I guess this is a security feature, since a process cannot even listen on some ports without having elevated privileges. I wouldn't expect another process being able to capture all this traffic anyway. This would also require a mechanism of sending the same stream to multiple processes (TCP listeners and all-protocol listeners).

But I didn't even know it was possible to capture traffic from multiple transport layer protocols using a syscall, perhaps that syscall requires elevated privileges itself..?

  • Hawzen 3 hours ago

    > perhaps that syscall requires elevated privileges itself..?

    You are exactly right

pjmlp an hour ago

We would be using one of the other protocols that were common at the time, before TCP/UDP/IP took over everything.

globular-toast 8 hours ago

What you want is a packet socket:

   sock_raw = socket(AF_PACKET , SOCK_RAW , htons(ETH_P_ALL));
IP networks should forward anything, but NAT is a major problem. Would be interesting to try with IPv6.
  • ay 7 hours ago

    To second this and expand on the reasons behind:

    The way the NATs (network address translators) are sharing the scarce public IPv4 addresses is by multiplexing on the transport level fields (ports in case of TCP/UDP and IDs or inner packet transport level fields in case of ICMP).

    Since they are unaware of your protocol, they get into a “special case mode”, which on a naive translator might consume a whole IP address (so you would really make a network admin with a few of those, because you exhaust all their available addresses :-) ; but on the carrier grade NAT there are safeguards against it and the packets are just dropped.

    • elcritch 4 hours ago

      At some point it seems like we just need to start passing laws requiring (major) internet services to provide all services on IPv6. Routers could then intercept ipv4 only devices and their dns locally and translate it to ipv6.

      Still lame that in 2024 major services like Steam and Quest basically require IPv4.

      I want to be able to use the internet without silly things like exhausting a network admins ipv4s.

    • geokon 7 hours ago

      This is outside my area of expertise.. so naiive question.. but ports aren't tied to the protocol .. right? If you open a raw socket, it's still on some associated port number. NAT traversal multiplexes ports.. so why would that preclude using any arbitrary protocol?

      • tsimionescu 7 hours ago

        Ports are very much a concept of the transport layer. They are a very useful concept, so they are used in all major transport-layer protocols, but they are not necessary in a theoretical sense (though a transport layer protocol without them would only allow a single stream of traffic between any two machines, or at least IPs). But TCP port 22 is a completely different thing than UDP port 22, and they are both completely different from SCTP port 22. To prevent confusion, IANA typically assigns the same port number for a protocol on both TCP and UDP (e.g. DNS over TCP uses TCP port 53, just like DNS over UDP uses UDP port 53; and QUIC uses UDP port 443, just like HTTPS uses TCP port 443).

        When a machine receives a packet, after the Ethernet and IP layers have made sure the packet is addressed to this machine, the very next thing that happens is checking what transport layer implementation should receive the packet - and this is done based on the "transport" bits in the IP header. If the packet is, say, TCP, then the TCP layer starts reading its own header and finds out the port number, and proceeds from there.

        • geokon 2 hours ago

          > But TCP port 22 is a completely different thing than UDP port 22

          OH woah woah. Okay that's the root of my confusion then. Nevermind. I thought the OS network stack exposes ports and packets come on to ports

          Nevermind then :)

      • numpad0 6 hours ago

        TCP/IP is a protocol called TCP encapsulated in protocol called IP. Raw IP packets have "protocol number" for use by payload, but not port number. Port number is technically part of custom data that IP layer should not care about.

        If you mean socket as in `my_socket = socket(AF_INET, SOCK_STREAM, 0);` that's TCP/IP, not raw IP, so it will have port numbers in the TCP part of the packet. `SOCK_STREAM` and `SOCK_DGRAM` respectively correspond to TCP and UDP. Raw IP sockets on Linux can be created by `socket(AF_INET, SOCK_RAW, protocol);` and that will have no TCP/UDP header attached by the Kernel after the IP header in the packet.

        (googling a bit as I write, forgive errors)

      • Hikikomori 3 hours ago

        IP has no ports, it has a protocol field, TCP, UDP, IPsec and other layer 4 protocols have their own protocol number. NAT (PAT really) uses TCP and UDP ports to connect what can be considered separate connections, ie a connection on the inside and a connection on the outside, these connections don't even have the same source/destination ports as connection state tracking is what connects these separate connections.

        The socket API has different types of sockets, typically you use stream (TCP) or dgram (UDP), but you can also get a completely raw socket where you need to construct your own ethernet and IP headers, or create your own replacement of IP protocol. So socket does not mean only TCP/UDP or something with ports, unix sockets are another example of this.

      • nlitened 7 hours ago

        Ports are part of TCP and UDP, not IP, as far as I can tell. So ports are tied to the protocol

      • withinboredom 7 hours ago

        Ports are tied to the protocol. IP layer doesn’t have ports (it’s in the name).

        • pta2002 6 hours ago

          I've been trying for a few minutes to figure out what you mean by "it's in the name"... What about Internet Protocol implies that ports are not an inherent property of it?

          All I can think of is that an "IP address" does not have a port component and that's all that IP deals with.

          • withinboredom 5 hours ago

            yep. That is exactly it. The IP layer is specifically a map with destination ip address and source ip address. The port is not included.

      • globular-toast 5 hours ago

        When using a packet socket you are sending bytes directly to the device driver. The only ports at this level are the physical ports on your machine which will have names like "eth1" etc. Assuming an Ethernet driver, the bytes must be a valid Ethernet frame[0], but that's all. The payload is just bytes.

        [0] https://en.wikipedia.org/wiki/Ethernet_frame

  • the8472 5 hours ago

    IPv6 alone won't help if there are some firewalls sitting between the endpoints that will drop anything they don't know. Having two linux hosts without firewalls talk to each other over IPv6 without consumer grade routers (which tend to come with firewalls by default) between them might work.

  • Hawzen 8 hours ago

    That's very interesting!

    I'll try out IPv6

    • globular-toast 4 hours ago

      If you want to play with packet sockets you might find my notebook useful where I was going to implement TCP/IP from scratch (well, from layer 2 up): https://github.com/georgek/notebooks/blob/master/internet.ip... I did ICMP over IP but got bored before I got to TCP, though (it's way more complicated than IP). You could drop in your custom protocol in place of ICMP.

MisterTea 2 hours ago

There's also Bell Labs IL - Internet Link - used to speed up 9P links for Plan 9 over LANs. https://doc.cat-v.org/plan_9/4th_edition/papers/il/

And this brings me to why I love networking on Plan 9. First off the dial string, net!address!service, passes the network along with the address and service port letting you easily switch protocols as needed. e.g. a program could listen on IL port 666 using the dial string il!*!666 and the client dials il!10.1.1.1!666. Second, that lets you dial and listen on any protocol from any program. If one wanted to use raw Ethernet you use the dial string ether0!aabbccddeeff. If I wanted to add a protocol like quic or a dead one like ipx/spx I just need to mount it to /net and keep the semantics the same as other services in net and any program can dial that protocol - ezpz user space networking stacks. Powerful abstraction.

casey2 4 hours ago

It's weird to me that the definition of portable has changed from "can be moved" to "can be moved to another machine" to "can be moved to another environment" to "can be moved to another program"

We had the option to make every object addressable and we chose not to. Why keep sweating over it, just accept that software only works on the devs machine and ship that.

jmclnx 2 hours ago

We would all be on UUCP. That was doing similar things before TCP/UDP.