the transformer model used by LLMs is a pretty simple approach that works exactly for the purpose of its proprietors. it's a keyword search engine with an obfuscation layer attached. it is the perfect expression of what the tech sector gets all that damn money for. it's not to make things more efficient. and it identifies information architecture—library science—as the hero to fight the final boss of capital
0 Boosts
0 Likes
@hipsterelectron This doesn't quite seem right to my ears: once you hit the Pareto-optimal frontier, efficiency and resiliency are fundamentally at odds with each other.
For example, consider sorting lists of elements where the comparison function has some probability of messing up, say 1 in 100 or 1 in 1000 or whatever, and assume the probability that any given execution of the comparison routine being faulty is independent of all other executions.
Quicksort is very efficient, with asymptotic complexity at the theoretical maximum, but it's disastrous with unreliable comparison functions.
Bubblesort is often used as an example of an inefficient sorting algorithm, but it performs as well as can be expected (in terms of correctness, not necessarily time) in the presence of faulty comparisons. Unlike with quicksort, a bubblesort will approach a nearly-sorted list, with every element being almost certain to be within one or two elements of their correct positions.
What you describe sounds likely to lead to a resilient system, but not necessarily the utmost efficiency.
In my mind "efficiency" is approximately "reward divided by cost", though less careful use of the word can mean a number of other things.
So I'd agree that efficiency is closer to throughput than latency, but not really the same either because it's fairly common that the most efficient processes tend to have difficulties with throughput, and processes that provide great throughput to kind of suck in terms of efficency.
In my mind, conflating efficiency with throughput is similar to conflating efficiency and power.
@leon_p_smith you mention "faulty comparisons" but i'm not familiar with a model of faulty comparisons (which includes a hidden distributional assumption) and i would absolutely challenge your argument for its optimality
@hipsterelectron The exact distributional model doesn't much matter to the point I'm trying to make in this case, which is once something reaches a certain level of "competence", efficiency and resiliency are fundamentally in tension with each other.
The intention for the faulty comparisons example is that every time the comparison subroutine is executed on the computer, independent dice are rolled to see if the result is going to be inverted or not. But variations on this should often work just as well, as long as you don't get into comparison subroutines that consistently return the wrong result on certain pairs of inputs.
Also, you usually are never quite certain if you are on a Pareto-optimal frontier. More often it's a comparison between multiple things, some of which might be on the Pareto frontier, and others are more "obviously" not on the Pareto frontier (inferred from the relationship to the things that *might* be on the frontier)
So I'm not going to make any claims that bubblesort is on the relevant Pareto frontier, but it's not actually all that hard to see that it is "optimal" in the sense of producing a list that is (almost) in the right order. (After all, a comparison function that's right 99% of the time doesn't have any reasonable chance of sorting a list of 1000 elements...)
Similarly I'm not going to claim that Quicksort's on the Pareto frontier, but it's well known that it is asymptotically optimal in terms of the number of comparisons made.
The reason is that you count the number of orderings of a list, and then take the binary logarithm of that, and log(n!) = O(n*log(n))
It's not difficult to see that a sorting routine that uses the absolute minimum number of comparisons to sort the list cannot be resilient. If you think of the result of each comparison as being an input to a function that permutes the list, an optimal number of comparisons means you have zero error correcting bits left over.
Of course quicksort is only *asymptotically* optimal in this sense, not necessarily optimal, but it's not hard to actually run this experiment yourself and see that yeah, when your comparison function exhibits this kind of error model, quicksort doesn't do well. You get badly broken outputs, not nearly-sorted lists.
But you also don't need to know that anything is on a Pareto frontier to be able to use a couple of examples to understand that the Pareto frontier does indeed *exist*.
@leon_p_smith when we speak of the optimality of sorting functions in complexity theoretic terms, we assume a basic cost model to operations such as comparison. if we seek to modify the comparison operation to account for failure, we must then consider that failure typically in real terms is not a simple decontextualized binary event, but something with a more complex measurement apparatus applied to it, particularly since failure is something that interrupts and indeed invalidates the unit cost assumptions applicable to complexity theoretic analysis.
probability theory then comes into play to codify observed events and infer the state of the system. this interplay between complexity theory and probability (or "information") theory is a central debate in the field of cryptography, in which complexity theoretic assumptions are used to describe an imposed cost model, while probabilistic noise is capable of jumping the gap from exponential to undecidable problems, as you may be familiar with from the one-time pad.
randomness generation (and other related operations) in cryptography must seek to—at the same time—resist complexity-theoretic and probabilistic cryptanalysis. this is in fact possible, but it gets to the core of modern knowledge of secrecy. so cryptographic protocols are often framed as a "game" in order to model the interactive nature of exploratory analysis in an online system.
what leads us onwards into infinity here is the reactivity to events and behavior, which generalize the static distributional assumptions common to simple statistics and "information" theory. this is a stateful process—the single bit of information leaked per failure is stateless, which is why it's trivially decidable. it's also not realistic, as no measurement is truly stateless, because every measurement makes a statement not just about itself but about the entire set of observations, and failing to ever update an inferred distribution after observing an event is a trivially suboptimal strategy.
if you begin with the assumption of independent (stateless) dice rolls (which is a hidden source of infinity introduced into a finite system), and to construct a system that assumes the existence of such rolls, then you can get something without undecidability, which is to say, without an infinity. but you most certainly cannot introduce independent dice rolls into a finite system and expect to apply simple complexity theoretic analysis, which is not just a finite system but a deterministic one without hidden state.
i mean, you can, but i don't really accept it as a useful problem. typically game theory has the concept of distinct equilibria instead of optimality, where optimality is not global but given a strategy, which can be observed but not known in advance. and i'm not terribly such a big fan of incorporating concepts like pareto optimality from economics into this space anyway, since i find the swiss bank nobel prize even more of an attempt to frustrate real scientific endeavor than the regular nobel prizes
making things more efficient is antifascist because fascism is parasitic and ideologically opposed to efficiency. the fascist claim to efficiency is that 6 million jewish people were dead weight upon society. efficiency, as any graph theorist can tell you, arises from highly connected networks with lots of redundancy—this is not a quality of the internet