If that occurs and it’s a substantial enough body of output that it is itself copyrightable and not covered by fair use. Confluence of those conditions is intentionally rare.
Deployments like bedrock have no where near SOTA operational efficiency, 1-2 OOM behind. The hardware is much closer, but pipeline, schedule, cache, recomposition, routing etc optimizations blow naive end to end architectures out of the water.
Many techniques are documented in papers, particularly those coming out of the Asian teams. I know of work going on in western providers that is similarly advanced. In short, read the papers.
Look, forget the details, step back and consider the implications of the principle.
Someone should not be able to write a semi-common core utility, provide it as a public good, abandon it for over a decade, and yet continue to hold the rest of the world hostage just because of provenance. That’s a trap and it’s not in any public interest.
The true value of these things only comes from use. The extreme positions for ideals might be nice at times, but for example we still don’t have public access to printer firmware. Most of this ideology has failed in key originating goals and continues to cause headaches.
If we’re going to share, share.
If you don’t want to share, don’t.
But let’s not setup terminal traps, no one benefits from that.
If we flip this back around though, shouldn’t this all be MPL and Netscape communications? (Edit: turns out they had an argument about that in the past on their own issue tracker: https://github.com/chardet/chardet/issues/36)
LGPL applies to the LGPL’d code, not to every piece of code someone might add to the repository or under the same name implicitly.
The claim being made is that because some prior implementation was licensed one way, all other implementations must also be licensed as such.
AIUI the code has provenance in Netscape, prior to the chardet library, and the Netscape code has provenance in academic literature.
Now the question of what constitutes a rewrite is complex, and maybe somewhat more complex with the AI involvement, but if we take the current maintainers story as honest they almost certainly passed the bar of independence for the code.
It’s not a good idea and that’s where I’d really start with the dated commentary here rather than focusing on the polling mechanism. It depends on the application but if the buffers are large (>=64kb) such as a common TCP workload then uring won’t necessarily help that much. You’ll gain a lot of scalability regardless of polling mechanism by making sure you can utilize rss and xss optimizations.
It's been a while but why is uring not helpful for larger buffers? I'd think the zero-copy I/O capabilities would make it more helpful for larger payloads, not less
uring supports zero-copy, but is not a copy-reduction mechanism; it is a syscall-reduction mechanism. Large buffers mean less syscalls to start with, so less benefit.
Exactly this. The kernel alloc’d buffers can help but if that was a primary concern you’re in driver territory. Anything still userspace kind of optimization domain the portion of syscalls for large buffers in a buffered flow is heavily amortized and not overly relevant.
The human driver of the project has a comment that is reporting that the project has no structural overlap as analyzed by a plagarism analysis tool. Were comments excluded from that analysis? Is your comment here based on the data in the repo?
I agree with you, but also we will start sharing these conversation traces more and more. That's why it is important for redaction to be in the export pipeline. There can be both deterministic (eg regex) and LLM-based redaction.
> You can already configure your initial congestion window, and if you are connecting to a system expecting the use of PQ encryption, you should set your initial congestion window to be large enough for the certificate; doing otherwise is height of incompetence and should be fixed.
The aggressive tone is no defense against practical problems such as the poor scalability of such a solution.
> You could also use better protocols like QUIC which has a independently flow controlled crypto stream and you can avoid amplification attacks by pre-sending adequate amounts of data to stop amplification prevention from activating.
Not before key exchange it doesn't. There's no magic bullet here.
A refresher on the state of TFO and QUIC PMTU might be worthwhile here before jumping this far ahead.
You have asserted without evidence that the increased certificate chain size is the primary scaling bottleneck. I assert that the bottleneck is most likely due to accidental complexity elsewhere on the argument that claimed problems look to be far in excess of the essential complexity.
> Not before key exchange it doesn't. There's no magic bullet here.
I was incorrect. Rereading the QUIC standard I see that they do not flow control the CRYPTO packet number space/stream. I thought they did because it is so easy to do that I did it as a afterthought. Truly another example of fundamental design errors introducing accidental complexity that should be fixed instead of papered over.
Can you elaborate a bit more about what you think the unnecessary complexity here?
A basic source of concern here is whether it's safe for the server to use an initial congestion window large enough to handle the entire PQ certificate chain without having an unacceptable risk of congestion collapse or other negative consequences. This is a fairly complicated question of network dynamics and the interaction of a bunch of different potentially machines sharing the same network resources, and is largely independent of the network protocol in use (QUIC versus TCP). It's possible that IW20 (or whatever) is fine, but it may well may not be.
There are two secondary issues:
1. Whether the certificate chain is consuming an unacceptable fraction of total bandwidth. I agree that this is less likely for many network flows, but as noted above, there are some flows where it is a large fraction of the total.
2. Potential additional latency introduced by packet loss and the necessary round trip. Every additional packet increases the chance of one of them being lost and you
need the entire certificate chain.
It seems you disagree about the importance of these issues, which is an understandable position, but where you're losing me is that you seem to be attributing this to the design of the protocols we're using. Can you explain further how you think (for instance) QUIC could be different that would ameliorate these issues?
For point 1, as I noted here [1], total bandwidth and resources are dominated by large flows. Endpoints are powerful enough to handle these large flows. The primary problems would lie with poor intervening networks and setup overhead.
For point 2, that is a valid concern of any case where you have just plain old more data. This dovetails into my actual point.
The problem of going from a 4 KB certificate chain to a 16 KB certificate chain, 160 KB certificate chain, or any arbitrary sized certificate chain should be equivalent to the problem of "server sends N byte response like normal". To simplify the problem a little it is just: the client sends a R-byte request message, the server responds with the Q-byte response message (which happens to be a certificate chain), the client sends the P-byte actual request, the server responds with a K-byte response message. So, at the risk of over-simplification, the problem should only be marginally harder than any generic "time to Q + K bytes".
Of course, if you previously had a 4 KB actual response and a 4 KB certificate chain and now it is a 160 KB certificate chain, you are going from "time to 8 KB" to "time to 164 KB". That is the essential complexity to the problem. But as I noted in my response to your point 1, the amount of server and client resources actually being expended on "small" requests is small with only poor networks where you are now consuming significantly increased bandwidth being a problem.
This then leads into the question of why "time to 8 KB" versus "time to 164 KB" is viewed as such a dramatic difference. This is a artifact of poor protocol design.
From a network perspective, the things that mostly matter are end-to-end bandwidth, end-to-end latency, endpoint receive buffer size, and per-hop bandwidth/buffering. You have a transport channel with unknown, dynamic bandwidth and unknown latency and your protocol attempts to discover the true transport channel parameters. Furthermore, excessive usage degrades overall network performance, so you want to avoid over-saturating the network during your discovery. In a ideal world, you would infer the transport parameters of every hop along your path to determine your holistic end-to-end transport channel parameters. This is problematic due to paths shifting or just plain dynamic throttling, so you will probably only limit yourself to "client to common bottleneck (e.g. your router) path" and "common bottleneck to server path". The "client to common bottleneck path" is likely client controlled and can be safely divided and allocated by the client. The "common bottleneck to server path" is not efficiently controllable by the client so requires safe discovery/inference.
The "initial congestion window" is a initial bandwidth-delay product to avoid over-saturating the network. This does not directly map to the transport parameters that matter. What you actually want is a initial safe "end-to-end bandwidth" which you refine via the discovery process. The latency of your roundtrip then only matters if the endpoint receive buffer size is too small and only effects how quickly you can refine/increase the computed safe "end-to-end" bandwidth.
Under the assumption that a 16 KB "initial congestion window" is fine and we assume the default RTT is ~100 ms (a somewhat reasonable assumption for geographically distributed servers who want to minimize latency) then that is actually a initial safe "end-to-end bandwidth" assumption of (16 KB / 0.1 s * 8 B/b) = ~1.3 Mb/s. Assuming the client advertises a receive buffer large enough for the entire certificate chain (which it absolutely should) and there are no packet losses, the client would get the entire certificate chain in ~(1 s + RTT) in the worst case. Note how that has only a minor dependency on the end-to-end latency. Of course it could get the data sooner if the bandwidth gets refined to a higher number, and a lower RTT gives more opportunities to get refined to a higher number, but that bounds our worst case (assuming no packet loss) to something that is not really that bad especially for the poor network throughput that we are assuming.
This then makes it obvious how to improve this scheme by choosing better initial estimates of "end-to-end" bandwidth or actively communicating that information back and forth. The "client to common bottleneck path" can be "controlled" by the client, so it can allocate bandwidth amongst all of its connections and it can set aside bandwidth on that leg for receiving. This allows higher initial "end-to-end" bandwidth assumptions that can be safely clipped when the client realizes it is in bad network conditions such as plane wifi. If the server determines "I have set aside N b/s to the 'internet' for this client" and the client determines "I have set aside M b/s from the 'internet' for this server" then your only problem is if there is a bottleneck in the broader backbone connections between the server and client. You would almost certainly be able to support better initial bandwidth assumptions or at least faster convergence after first RTT if you communicated that information both ways. This is just a example of what and how things could be improved with fairly minimal changes.
And this all assumes that we are even trying to tackle this fairly fundamental root issue rather than what are probably heaps of other forms of accidental complexity like middleboxes just giving up if the certificates are too large or whatever else nonsense there is which is what I am pretty sure is the real impetus by why they want the networking equivalent of wanting the 737-MAX to handle the same as a 737.
It's difficult for us to maintain documentation of exactly the kind you'd want there, though we do try to keep up with docs as best we can. In particular there is a fairly wide array of heuristics in the client to adapt to the environment that it's running in - and this is most true on Linux where there are far far too many different configuration patterns and duplicate subsystems (example: https://tailscale.com/blog/sisyphean-dns-client-linux).
To try and take a general poke at the question in more of the context you leave at the end:
- We use rule based routing to try to dodge arbitrary order conflicts in the routing tables.
- We install our rules with high priority because traffic intended for the tailnet hitting non-tailscale interfaces is typically undesirable (it's often plain text).
- We integrate with systemd-resolved _by preference_ on Linux if it is present, so that if you're using cgroup/namepsace features (containers, sandbox runtimes, etc etc) then this provides the expected dns/interface pairings. If we can't find systemd-resolved we fall back to modifying /etc/resolv.conf, which is unavoidably an area of conflict on such systems (on macos and windows they have more broadly standard solutions we can use instead, modulo other platform details).
- We support integration with both iptables and nftables (the latter is behind manual configuration currently due to slightly less broad standardization, but is defaulted by heuristic on some distros/in some environments (like gokrazy, some containers)). In nftables we create our own tables, and just install jumps into the xtables conventional locations so as to be compatible with ufw, firewalld and so on.
- We do our best in tailscaled's sshd to implement login in a broadly compatible way, but again this is another of those places the linux ecosystem lacks standards and there's a ton of distro variation right now (freedesktops concerns start at a higher level so they haven't driven standardization, everyone else like openssh have their own pile of best-guesses, and distros go ham with patches).
- We need a 1360 byte MTU path to peers for full support/stability. Our inner/interface MTU is 1280, the minimum MTU for IPv6, once packed in WireGuard and outer IPv6, that's 1360.
I can't answer directly based on "very custom" if there will be any challenges to deal with. We do offer support to work through these things though, and have helped some users with fairly exotic setups.
This problem space is not small enough to stay within current LLM attention span. A sufficiently good agent setup might be able to help maintain docs somewhat through changes, but organizing them in an approachable way covering all the heuristics spread across so many places and external systems with a huge amount of time and versioning multivariate factors is hugely troublesome for current LLM capabilities. They're better at simpler problems, like typing the code.
hurd init is a lot like systemd architecturally, it just gets to use kernel provided ipc rather than having to manage its own. if your objection to systemd is its architecture you don't want anything to do with hurd.
reply