Will Hoover

Week of April 14 – 21 2024

I hope everyone has had a wonderful week; this week was especially rewarding to be able to visually see the first steps the model is starting to take.

generated face from 5 epochs of training on full 70,000 image Flickr Face HQ dataset

With the training dataset that we are starting with being 70k images at 128*128 the training time has increased from the training set from 30 seconds per epoch to 1 hour and 25 minutes. A bit of a increase but with that the difference in quality and the amount of images the model was exposed to the additional training time is well anticipated.

picture from 150 epochs from a 1000 image training dataset

Now that the training is underway, I have been looking more into exporting the model into ONNX format, which has proven to be more of a hurdle than originally expected. The math that is being done in the mapping network is not fitting the ONNX format the way I was hoping it would. I’m sure there is a workaround for this, and I have been looking for it, but in the meantime so the team can get up and running with our generator and discriminator I have created checkpoint.pth files that copy all the relevant settings and weights so the model training can be started again in the future as a bit of a safety feature to keep progress every few epochs, and so that we can create a API for HuggingFace that will call one of our .pth files to load the necessary settings to create the prompted image.

Once the network is creating images, we are comfortable with then we will start training the model on a database with a description for each of the images, that way our generator will then be able to understand what each individual thing is so that it can create images and know what it is creating.

The training is going well, though we have many days of training to go before we get the results that we are looking for.

Thanks for taking your time with me this week!

William Hoover