In my quest to get AI applications depend less on LLMs, I came up with an idea: Let the AI tell us how it thinks, so that its secrets can be extracted and redone with classical logic.
prompt:
> Whenever an unforeseen situation happens, write in a log, using symbolic reasoning, the event, what hypotheses you came up with, what evidence would (dis)prove them, and what decisions you took.
And the logs are... interesting. 
Then I asked the AI to generalize the rule so the logs could be more detailed.
Now the logs explain tons and tons of decisions on debugging and refactoring code, complete with their boolean syntax and nomenclature.
This is invaluable info!