I’m a student that replicated OpenAI’s GPT2–1.5B.
I plan on releasing it on the 1st of July.
I’m waiting because I want to give people time to convince me I’m wrong if I missed something.
OpenAI decided not to release the full scale version of their model (1.5B), because they were afraid of its potential security implications, in particular in generating fake news. They did release a smaller version (117M) and later a middle sized version (345M). In theory, there was no significant barrier to scaling up 117M or 345M to 1.5B, but there was a practical one: Compute. The estimated cost to create 1.5B is around 40k$ in cloud computing.
The decision to not release their model into the open, despite the “Open” in their name, garnered a wide range of responses. But to this day, 1.5B remains unreleased (except to a few research partners) and, to my knowledge, unreplicated. No individual or reasonable academic research group would have access to enough resources to create 1.5B from scratch.
Well, I replicated 1.5B.
I did it because, dammit, it was cool!
I’ve experimented quite extensively with the smaller models, and have had great fun with it. Me and friends have had literal hours of fun trying to get the thing to generate funny texts for us to read aloud to each other (We’ve found it makes especially hilarious religious rants if you feed it with biblical quotes). Me and a friend have been developing a video game incorporating GPT2 and even once during a party we sat down around a piano and prompted the AI until it spit out something resembling lyrics and so my buddy Sebastian turned it into an impromptu space opera musical.
Sometimes, while feverishly hacking away at my latest AI abomination (GPT2 is not my weirdest project, believe me), I pause for a moment, cyberpunk synthwave music blaring from my headphones, and wonder to myself: “Am I a Black Mirror character?”
I’m not working under the direction of any government, or university, or large corporation. I’m just a curious undergrad student that spends his free time experimenting with AIs instead of going outside and talking to girls.
And I wasn’t paid a cent for any of it. No one told me to do this, no one supported me in doing this, I could have spent that time with my friends or studying for university or doing literally anything else.
https://medium.com/@NPCollapse/gpt2-cou ... 3c6639a3a8