Post by @jcoglan@mastodon.social

RE: https://hachyderm.io/@mjk/117541836725953937

this gets at what I've been trying to articulate for a while: taking the stochastic output of a generator and then filtering it through some mixture of automated tests and your own judgement is a deeply flawed method of making software compared to *constructing* it from things with known properties

Matt Keeter

@mjk@hachyderm.io 20 hr. ago

this is something that I've been struggling with – there are now a half-dozen new CAD kernels in Rust, all with the same perfectly polished websites and bold claims about performance, test coverage, and quality

(from https://jepsen.io/analyses/particledb-1.1.4)

4 Boosts

0 Likes

This federated comment is not yet available here.

jcoglan

@jcoglan@mastodon.social 2 hr. ago

e.g. most of the content of https://escodb.github.io/design/ -- and the entire value to me of having written is -- is discovering what shape the internal building blocks of escodb must have in order to meet the intended design goals

if you handed me a fresh impl of escodb (however it was produced) I would have an extremely hard time determining whether it met the required properties to work correctly. whereas I know my version works *because of the methods used in designing and building it*

Introduction - EscoDB Internal Design

Anthony

@abucci@buc.ci 9 min. ago

@jcoglan@mastodon.social You've read Peter Naur's Programming as Theory Building? He'd be happy to read what you're saying here.

Thanks for this thread, and sorry for spamming your mentions. It's refreshing to read.

jcoglan

@jcoglan@mastodon.social 2 hr. ago

databases are a good example of software where the correctness depends on a lot of factors that are not easily observable externally. you cannot build software like this by filtering random output through a set of e2e tests. you *have* to design the internals very carefully

jcoglan

@jcoglan@mastodon.social 2 hr. ago

I saw a thread earlier repeating the lie that the only thing that matters about code you write is that it "does the thing". trouble is, an adequate description of "does the thing" in many cases is so complicated that most people are not capable of writing an automated representation of it

2 more replies

jcoglan

@jcoglan@mastodon.social 2 hr. ago

I have done a modicum of database programming and other things where correctness under concurrent execution with partial failure, as well as performance, is very important. you mostly do this by reasoning very carefully about the shapes of the abstractions you're building with and how they interact

jcoglan

@jcoglan@mastodon.social 2 hr. ago

*constructing* a system by breaking it into smaller pieces, thinking really carefully about how those things should behave, and additively building the system up, is in my opinion far easier than having a whole impl dumped on you and trying to look for flaws in it

Anthony

@abucci@buc.ci 13 min. ago

@jcoglan@mastodon.social I've been babbling about this for awhile here, that the distinction you're drawing mirrors that between gradient ascent vs generate-and-test in the world of optimization. Just to be clear about what I mean by that: generate and test is as it sounds. Rpeatedly, usually stochastically, generate potential solutions and then use the tests to determine which to accept or reject. Gradient ascent means you have some notion of which way is up and you follow that. If you have such a gradient you can use it with generate-and-test, but it'll be orders of magnitude slower and often will yield inferior results too.

A competent software developer performs gradient ascent, as you've pointed out. LLMs with acceptance tests perform generate-and-test (to a first approximation anyway). The null hypothesis is plain enough.