Ok, the current plan is to sample \(15 * 15\) vectors from the t-SNE reduced 2D latent space. Get the actual encoder generated feature tensor for that point in the latent space. Decode the feature vector. Put the decoded data/image in the appropriate spot in a large numpy array. That array will be \((15 * 28) \mbox{ x } (15 * 28)\) (176,400 float64 elements/pixels). Let’s hope I have enough memory.
Generative Model Visualization
I will be putting the code for this effort in a new if block. And, I think I am going to use the latent space generated by the training dataset. I will start by getting the data and selecting the points we are interested in. Then I will create the empty figure array/grid and instantiate the ImageGrid class to help sort and keep track of things. Then we will loop through the indices for the relavent t-SNE data. Getting or generating the data we need to build the visualization. Let’s get that bit done as it mostly repeats previous code.
I am only running it for 3 images while I work on the code. Want to make sure each step works before I go all in.
... ...
do_dev = True
# let's generate the visualization
# instantiate model for jitscript file
fl_pth = cfg.sv_dir/"AE_Simple_jitscript_32_4.pt"
aec = torch.jit.load(fl_pth).to(cfg.device)
aec.eval()
# grid and image sizes, used within the if block
n_w, n_h = 15, 15
d_sz = 28
# create array with which to build the visualization
figure = np.zeros((d_sz * n_h, d_sz * n_w))
# instantiate the ImageGrid
g_ls = ae_utils.ImageGrid((-15, 15), (-60, -30), g_wd=n_w, g_ht=n_h, lbls=True)
for cnt, l_ndx in enumerate(slct_ls):
x, y = ls_data['tsne'][l_ndx]
# get row, column from ImageGrid for current latent space points
rw, cl = g_ls.get_grid_cell(x, y, l_lbl=ls_data["labels"][l_ndx].item())
if do_dev:
print(f"\n({x}, {y}) -> ({rw}, {cl})")
# get label for current latent space index
lbl = ls_data["labels"][l_ndx].item()
# load the related full feature tensor to the GPU
f_vector = ls_data["latent_spc"][l_ndx].to(cfg.device)
with torch.no_grad():
# only need the decoding for the current feature tensor
d_gen = aec.decoder(f_vector)
if do_dev:
print(lbl, "->", f_vector)
imgplot = plt.imshow(d_gen.cpu().reshape(d_sz, d_sz))
plt.show()
if cnt >= 2:
break
And the terminal output from our little test.
(mclp-3.12) PS F:\learn\mcl_pytorch\proj7> python autoe.py -rn rek_2 -bs 32 -ep 5
{'run_nm': 'rek_2', 'dataset_nm': 'no_nm', 'sv_img_cyc': 150, 'sv_chk_cyc': 50, 'resume': False, 'start_ep': 0, 'epochs': 5, 'batch_sz': 32, 'num_res_blks': 9, 'x_disc': 1, 'x_genr': 1, 'x_eps': 0, 'use_lrs': False, 'lrs_unit': 'batch', 'lrs_eps': 5, 'lrs_init': 0.01, 'lrs_steps': 25, 'lrs_wmup': 0}
image and checkpoint directories created: runs\rek_2_img & runs\rek_2_sv
(-14.008818626403809, -57.747093200683594) -> (13, 0)
2 -> tensor([-1.3894, -6.2014, 2.7763, 6.3715, -0.3869, 0.9281, -2.0430, -1.2327,
3.9075, -1.6893, -1.5129, 0.1337, -0.7059, -3.2612, -6.3269, 4.0852],
device='cuda:1')
(1.6075557470321655, -31.21147918701172) -> (0, 8)
3 -> tensor([ 3.3714, -4.6892, 0.5422, -7.6001, 4.6206, -2.7226, -1.3266, 4.2792,
0.6957, -4.0462, 2.1689, -1.8995, -1.0624, 5.1686, -7.8364, -6.2222],
device='cuda:1')
(13.68570327758789, -50.67435073852539) -> (10, 14)
5 -> tensor([-2.4617, -4.1799, -7.5049, -6.5646, 3.4770, -5.0908, -0.4671, 6.1720,
-7.2594, -2.4543, -2.2702, -1.2169, -3.2130, 3.1405, -1.2917, 0.9317],
device='cuda:1')
Not going to bother with the images. As I didn’t resize the plot window they were rather fuzzy. But as near as I could tell they matched the label for each processed index.
Time to build the image array. The first thing I need to sort are the array locations for each image in the figure array.
Given row, rw, and column, cl, values as returned by g_ls.get_grid_cell() we need to get the section of the figure array to which we will write/add the current image. We will need image size, d_sz, rows and columns at some location in the array. The rows will start at \(rw * d\_sz\) and end at \((rw + 1) * d\_sz\). Well, it will end at one less (array indexing does not include the last value). And of course the arithmetic will be similar for the columns. And, the section of the array will be accessed as follows.
figure[rw * d_sz : (rw + 1) * d_sz,
cl * d_sz : (cl + 1) * d_sz,
]
We will assign our current image (d_sz by d_sz) to that section of the array. Overwriting whatever is currently there. Hopefully just zeroes. Let’s get that set up. Then run the code for three sets of data and plot the result.
I am going to show a lot of the original code along with the changes. Mainly for context.
for cnt, l_ndx in enumerate(slct_ls):
x, y = ls_data['tsne'][l_ndx]
# get row, column from ImageGrid for current latent space points
rw, cl = g_ls.get_grid_cell(x, y, l_lbl=ls_data["labels"][l_ndx].item())
# if rw is None, we don't want to add anything to our image
if rw is None:
continue
if do_dev:
print(f"\n({x}, {y}) -> ({rw}, {cl})")
# get label for current latent space index
lbl = ls_data["labels"][l_ndx].item()
# load the related full feature tensor to the GPU
f_vector = ls_data["latent_spc"][l_ndx].to(cfg.device)
with torch.no_grad():
# only need the decoding for the current feature tensor
d_gen = aec.decoder(f_vector)
g_img = d_gen.cpu().reshape(d_sz, d_sz)
figure[rw * d_sz : (rw + 1) * d_sz,
cl * d_sz : (cl + 1) * d_sz,
] = g_img
if do_dev:
print(lbl, "->", f_vector)
# imgplot = plt.imshow(d_gen.cpu().reshape(d_sz, d_sz))
# plt.show()
if cnt >= 2:
break
# plot figure array and print grid
imgplot = plt.imshow(figure)
plt.show()
print("\n", g_ls.grid.astype(str))
Here’s the pertinent portion of the terminal output. Followed by the model visualation with the 3 generated images added to it.
[['' '' '' '' '' '' '' '' '3' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '5']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['2' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']]
They both look pretty similar. So, let’s give the whole space a go. But, there are 1,651 entries in the list of indices for our space. And, we only want to plot 225 of them. I planned to try and stop the loop at some point. But, the whole thing takes so little time to run, that I just processed all 1,651 t-SNE vectors.
And here’s the final grid and figure. I got rid of the axes in the final plot—they were meaningless for this situation. (A plt.axis('off') after the call to imshow took care of that.)
[['8' '' '8' '8' '8' '8' '' '3' '8' '' '' '' '5' '' '']
['8' '8' '8' '' '' '8' '' '3' '' '' '' '' '5' '5' '5']
['' '' '2' '8' '8' '' '' '' '3' '' '' '' '' '5' '']
['' '8' '' '8' '8' '' '' '' '' '' '5' '5' '' '' '']
['' '8' '' '' '8' '8' '' '1' '5' '' '6' '5' '5' '5' '']
['' '' '8' '8' '' '' '2' '' '' '' '5' '5' '5' '5' '5']
['2' '' '' '8' '' '' '8' '' '' '5' '5' '5' '5' '5' '5']
['' '' '2' '8' '' '8' '8' '' '5' '5' '5' '5' '5' '5' '5']
['' '2' '' '2' '' '2' '8' '' '5' '5' '' '5' '5' '' '5']
['8' '' '2' '2' '2' '' '' '' '' '5' '' '5' '' '' '']
['2' '8' '8' '' '' '' '' '' '5' '5' '5' '5' '5' '5' '5']
['2' '3' '2' '' '' '' '' '' '' '' '' '5' '5' '' '']
['' '' '2' '2' '' '' '' '' '' '5' '' '' '5' '5' '5']
['2' '' '' '2' '' '' '' '' '5' '5' '' '5' '' '5' '']
['2' '' '2' '2' '2' '' '' '' '' '5' '' '5' '' '5' '5']]
We should be seeing 8s, 2s and 5s. But some 3s have crept in. Which, given their similarity to 8s, does make some sense given the very limited training I did. And some 8s are encroaching on the 2s space.
That’s a Wrap
I am thinking I will call this post done. Though I am considering using a larger grid and plotting the whole generative model latent space. But most likely with the data saved for the test dataset. Much shorter loop.
That said, I enjoyed coding the various model plotting sub-projects. And, I think I learned a thing or two along the way.
May your experiments prove as satisfying and enlightening.
Addendum
I decided to plot the whole latent space for the generative model visualization. Well, the portion of it that will fit into a 30 by 30 grid. A bit of refactoring. Modified the loop to use the 2D reduced data for the full training data (60,000 points). Modified all instances of the index value from the set of indices for the reduced space to use the index value from enumerating the list of t-SNE data. Added code to make the image size bigger (12 x 12). The loop processed the whole list, including generating the images from the appropriate feature vector, in under a minute. And here it is.
