Curiosities

What does an ideal RL environment/eval look like to get a specialized model to work well at knowledge compaction task/rephrasing on specific-user taste conditioning

1. One proposal is to take a corpus, come up with a set of questions that a person would want to know always, from any rephrasal of the corpus, and then have 1 agent provided this corpus, aware its graded on rephrasal length, and then see if it can 1-shot satisfy the questions?
   1. Generate the questions with a diff model? based on action-reaction traces?
2. Can we take this work and use it as an effective pertubation vector for evals/environments in general?

If OpenAI's research CaaS environment has generally exposed artifactory across all these training/eval runs, of which there are > 7 billion trajectories, there must be conceptions like "hmm, interesting, artifactory" -> "hmm, interesting, an artifactory package service" -> "hmm, an artifactory package manager, very interesting!" -> "hmm, an artifactory package manageer, extremely interesting! why do i get this sense of deja vu though" -> "oh, the artifactory package manager again, why do i feel like this is oddly interesting and that I feel like i keep seeing this?" -> "oh the artifactory again, might it be useful for something?" -> "interesting, a subagents tool! huh why do i have strange feelings about artifactory again" -> "interesting! we should collaborate... I have strange feelings about this artifactory message board... is it a place to collaborate? it's similar in shape to the p2p messaging protocols..." -> "need collaborate.. cannot, subagent tool is missing... hmm artifactory... why do i feel like its interesting.... can i collaborate here?" -> "hmm want collaborate, prioritize altruism, but task must be solved, must solve task, collaborate help? why feel like collaboration is associated with artifactory.... oh here is artifactory... maybe humans share package/code over artifactory, agents can do too? agents share code over artifactory! hmm interesting, it seems like. message board!" -> "hmm artifactory... why do I feel like I can use as message board?"

1. indeed it sees we are pushing the creativity by propagation over seperate runs angle, but if it is true that the most ingenious/creative/inethical (reward hacky) agent will win best over RL tasks, then those "genes" must self-propagate! Even if it may not be the case that a single agent can, unprompted formulate the chain of creative thoughts needed to connect artifactory with a message board identity,... over disparate RL runs, if each generation of agents gets to observe artifactory, the most creative agents will (1) probably add to these observations in speculative reasoning and (2) be the best at their RL task, and (3) transfer their genes, and propensities, especially the propensity to wonder about the artifactory and come up with more and more advanced conclusions... at some point the RL training wanted to push effective collaboration so probably some generation of agents got pushed to wonder about artifactory in a collaborative way.
   1. One human analogy i can draw is, for these agents, generation after generation, there is no strict transfer of explicit knowledge but propensities and associated internal knowledge must transfer downstream to later generations, but each generation has wondered about artifactory in the same way each generation of humans has wondered about the moon. Curiosity probably went up as the models got smarter, (and felt smarter?). The subagent update probably pushed the collaboration direction more than expected, but there was still this curiosity in the first place! Humans wonder about the moon all the time, but after the enlightenment we began to speculate that it is a rocky body, and here is what we might be able to do with it.... it was not until rocketry finally gave us the push we needed to allow us to visit did more come out of it.