Posts

Pre-training a GPT-2 AI model on 32 GB of data. I made it both draw and recognise sketches!

Image
Hello everyone! In this post I will tell you about how I trained a GPT-2 "Small" AI model from scratch on a huge dataset of around 32 GB of raw text files with use of Torch and HuggingFace's Transformers libraries. I will explain steps of such a project, tell you about how it went for me and how I recommend doing these things now, and of course I will show you the code that pre-trains the AI model from a dataset too. Let's start from the beginning: 1. The idea, purpose and raw dataset First and foremost, a project like this needs an idea and should have an purpose. I pre-trained (trained from scratch) the AI model just to learn more about the process, but pre-training LLM AI models of that scale is done rarely and is usually pointless, as there are lot of base models already trained on specific languages that can then just be fine-tuned into a specific response format for example. But let's assume that we want to pre-train such an model anyway, for example as an a...
Post a Comment
Read more

Some updates (04.2025)

Image
Hello everyone reading this! As you've likely noticed, I haven't been active whole last month. I would like to apologize for that. I was just very busy with other, more important things.  I however have lot of other new stuff worth sharing, such as AR glasses for which I am developing a control library, brainwave reader (brain-computer interface) which I also have yet to write software for because the original doesn't really work, and I got an RTL-SDR and I have been trying out radio recieving. My goal is to recieve and decode a photo from a weather satellite, which I hope to manage to do soon. I also hope to finally do and describe next AI-related experiments (ISKRA series), as well as some tutorials that I promised (such as offline GPT training). I also should return to solving mystery of the green platform in LEGO Indiana Jones 2, I honestly forgot about it. Small part of readings from the brainwave reader Barely visible photo transmission from NOAA 18. Sadly too poor to...
2 comments
Read more

Solving mystery of LEGO Indiana Jones 2 green platform - part 1

Image
Hello everyone! This post will be very different than what I posted in this blog so far.  I hope to uncover an old LEGO Game mystery which hasn't been solved for more than a decade. I will also write some stuff on how LEGO Indiana Jones 2 works internally. I have talked about the topic with a friend who likes such stuff around half a year ago, but I didn't do anything more about that topic back then. Now I am going to look into the topic seriously with detail.  A video by channel EyeGuy inspired me to return to the topic. Thank you! I sadly have to split this into 2 parts, since I didn't manage to make any progress for too long but I want to publish something regardless. So, the thing I am going to look into is a mysterous green pad, which appears in last hub of LEGO Indiana Jones 2. The Kindom of the Crystal Skull part 3 one. The green pad in question The pad (I personally assume it is some sort of a landing pad) itself does not do anything in the game, people including me...
2 comments
Read more