ugh. I have some code I want to get done before a meeting in 30 minutes, and excitingly for my get-it-working-and-iterate-fast development style, it involves processing 13gb of JSON every iteration
this will not be fast
ugh. I have some code I want to get done before a meeting in 30 minutes, and excitingly for my get-it-working-and-iterate-fast development style, it involves processing 13gb of JSON every iteration
this will not be fast
it's actually worse than 13gb of json:
it's 13gb of JSONL-JSON turducken
that's JSON inside JSON which is in a JSONL
@foone Why though?
Why do that?
@NosirrahSec it's a local dump of a complicated database schema
oh god it's been 7 minutes and it's still running. the number of iterations I'm going to be able to do before this meeting is rapidly decreasing
okay it just finished
and failed completely
according to this all my 13gb of JSON has no keys in it.
seems unlikely
spot the obvious error:
all_seen_keys=set()
for row in rows:
all_seen_keys.union(row.keys())
print(all_seen_keys)
I should have done this before:
I made it only process the first 1000 rows, and iterated on that version, which finishes nigh-instantly. Then when it's working well enough, I turn off the limiter, and let it run on the full Xty-million-row file
which only breaks if there's weird edge cases that didn't exist in the first 1000 rows
(which only happens a few times)
@foone You can maybe go iteratively with increasing the limiter to 10k, 100k, ...
@foone@digipres.club finally, a real use case for one of those smd optimized json libraries like rapidjson
@puppygirlhornypost2 I wonder how much it'd even help here, my individual JSON documents are actually quite small and quick to parse, the problem is that I have many millions of them, so SMDing each one isn't going to give me much.
@puppygirlhornypost2 might be worth a try if I had more time though
distracting myself while it's running by opening up inkscape and designing some ITERATE FAST, EAT ASS stickers