Will Hoover

Week of 6/2 – 6/9/2024

This week we spent troubleshooting technical issues with the training computer. The laptop that the team was using to train the network was running Windows 11, with Ubuntu through WSL2. This has caused us issues since the beginning with Docker and training. We decided to switch the training laptop over to Linux Ubuntu, this way we can train the model without issue. We tested this setup on a already Ubuntu setup laptop, and we have not had any issues with training since. I set the old laptop up with Ubuntu and we are planning on doing a training switch soon, where Ethan will send me the .Pkl file he has been training on and I will pick up the training from the last Kimage we trained on.

Next week will be dedicated to training the model more, we have gotten it up to 1500 Kimage, which was the goal for training for the week. Next week I want to push that towards 3000-3500 Kimage, depending on how the TensorBoard training stats are looking. So far it’s not even close to leveling out for training so we should be good for a while. The images being generated were concerning at first, since it looked like all the generated images were following the same prompt, as it stands right now I think that’s a issue with lack of training, and when its more trained that will go away. There is some evidence for that, with the latest images being generated having different generated images than the earlier images.

We will have to wait and see, but hopefully next week we will have clearer images to look at

–         Will Hoover