Well, perhaps not surprisingly, I have decided to use that saved data to try and visualize the generative model. I.E. plot the decoded images over the latent space. Well sort of. Just a smallish grid with one digit per grid element. I will start with a subset of the complete latent space.
I had thought about just adding this to the last post. But having played around I had a number of issues due to a lack of thought and/or understanding. And, I think it will take a bit more planning and code than I originally thought.
Visualize Generative Model
There is a bit of a complication. The values plotted in the latent space are 2D. To decode a digit I need the 16 feature tensor matching that 2D latent space vector. To start I am looking at plotting the images for the range \((-15, -60)\) to \((15, -30)\). In a \(15\) by \(15\) grid.
So I need to get all the indices for values of the 2D latent space data within that range. Traverse the list selecting suitable values from the 2D space for each grid location. Then obtain the proper encoded tensor and decode it to get the image of the correct digit. Add that image to the grid. And finally plot the grid.
Get the Desired Indices
The t-SNE 2D data is in a Numpy array. So, Numpy syntax and methods/functions it is.
Let’s start by loading our saved data and having a look at it.
... ...
tst_gen_mdl = True # visualize the generative model (space)
... ...
if tst_gen_mdl:
# get saved latent space data from file
fl_nm = f"lspc_10k.pt"
fl_pth = cfg.sv_dir/fl_nm
ls_data = torch.load(fl_pth)
print(ls_data["tsne"].shape, type(ls_data["tsne"]), ls_data["tsne"][0])
ls_x_min, ls_x_max = min(ls_data["tsne"][:, 0]), max(ls_data["tsne"][:, 0])
ls_y_min, ls_y_max = min(ls_data["tsne"][:, 1]), max(ls_data["tsne"][:, 1])
print(ls_x_min, ls_x_max, ls_y_min, ls_y_max)
And in the terminal we get the following.
(mclp-3.12) PS F:\learn\mcl_pytorch\proj7> python autoe.py -rn rek_2 -bs 32 -ep 5
{'run_nm': 'rek_2', 'dataset_nm': 'no_nm', 'sv_img_cyc': 150, 'sv_chk_cyc': 50, 'resume': False, 'start_ep': 0, 'epochs': 5, 'batch_sz': 32, 'num_res_blks': 9, 'x_disc': 1, 'x_genr': 1, 'x_eps': 0, 'use_lrs': False, 'lrs_unit': 'batch', 'lrs_eps': 5, 'lrs_init': 0.01, 'lrs_steps': 25, 'lrs_wmup': 0}
image and checkpoint directories created: runs\rek_2_img & runs\rek_2_sv
(10000, 2) <class 'numpy.ndarray'> [-7.259082 -2.751589]
-79.921524 96.63328 -82.62013 75.35779
I had a lot of trouble and googling to get this sorted. But I think I finally have. A good part of it was not really not paying enough attention to the array shape and the correct Numpy indexing and/or selecting. A case of a little knowledge is a dangerous thing. And, I also was really not paying attention to the information I was printing to the terminal and what it was trying to tell me.
So, for quite some time, I was indexing the t-SNE data incorrectly. I expect the next bit of code took the best part of a day to sort out. With a bit more attention to detail, that likely could have taken significantly less time. Perhaps virtually none at all.
Just did a little test. One of the earliest things I tried (but not the first) was the following.
slct_ls = np.where((ls_data['tsne'][0] >= -15) & (ls_data['tsne'][0] <= 15) &
(ls_data['tsne'][1] >= -60) & (ls_data['tsne'][1] <= -30))
print("\nslct_ls:", type(slct_ls), len(slct_ls), slct_ls[0].shape, slct_ls[0])
Which gave me the following terminal output.
slct_ls: <class 'tuple'> 1 (0,) []
Do you see my error? I started messing around blindly. Reading search result after search result. After numerous attempts, I finally realized I was not indexing the array correctly. And, when I made a few small but immensely important changes to the above, I got the following.
slct_ls = np.where((ls_data['tsne'][:, 0] >= -15) & (ls_data['tsne'][:, 0] <= 15) &
(ls_data['tsne'][:, 1] >= -60) & (ls_data['tsne'][:, 1] <= -30))[0]
print("\nslct_ls:", type(slct_ls), len(slct_ls), slct_ls.shape, slct_ls[0:5])
print(ls_data["tsne"][slct_ls[0]], ls_data["tsne"][slct_ls[-1]])
print(ls_data["latent_spc"][slct_ls[0]])
And…
slct_ls: <class 'numpy.ndarray'> 331 (331,) [ 16 33 78 132 135]
[-11.234991 -54.738804] [ 6.832078 -31.907291]
tensor([ 0.2793, -0.8637, -4.8474, -0.6556, -3.0658, 3.2892, 11.1826, 4.2856,
8.0592, -3.6633, 0.0370, -1.1692, -1.7861, 0.2837, -1.3402, -1.2927])
That length of 331 matches the output of the final attempt that worked. And those two vectors are certainly within the selected range. Is that that the correct feature vector? I expect so. So, I think I have this part sorted.
Sorting Grid Location
Okay, we need to be able to determine the correct grid location for any of the t-SNE vectors. And, since we have 331 vectors and only 225 grid locations we need to not overwrite a grid location that already has an image. Well, guess we could, but…
I was thinking about a function. But because I don’t want to repeat some initializing of data every time the function is called, I am going to write a class to handle this work. It will be in the ae_utils module. Based on some parameters it will intialize a number of class variables and keep them in memory. A method call will sort out the grid location, returning its coordinates or None if it is already in use.
I plan on passing in the minimum and maximum values for the x and y axes of the grid and the number of cells in both directions. Those will be saved as class variables, along with a calculated divisor for each direction based on the supplied values. It will also initialize an array to store which grid locations have been filled. The latter assumes that if a grid location is returned that it will in fact be used. A potentially incorrect assumption. But since only me using the class, that’s okay for now.
I am not going to put any of the values related to this process in the config module. I will just include them in the current if block. We’ll start with the class. Then a test or two.
Because I am thinking I might want to record the labels when building the visualization, for comparision purposes, I am giving my self some options with respect to the type of grid and what gets stored in it. No explanation; I will let you sort that out from the code.
... ...
import math
... ...
class ImageGrid():
def __init__(self, x_lim, y_lim, g_wd=15, g_ht=15, lbls=False):
self.x_s, self.x_e = x_lim
self.y_s, self.y_e = y_lim
self.g_wd, self.g_ht = g_wd, g_ht
self.x_div = (self.x_e - self.x_s) / g_wd
self.y_div = (self.y_e - self.y_s) / g_ht
self.do_lbls = lbls
if self.do_lbls:
self.grid = np.zeros((g_ht, g_wd), dtype='S1')
else:
self.grid = np.zeros((g_ht, g_wd), dtype='uint8')
def __repr__(self):
return (
f"{type(self).__name__}"
f'((x_s, x_e)=({self.x_s}, {self.x_e}); '
f'(y_s, y_e)=({self.y_s}, {self.y_e}); '
f'(g_wd, g_ht)=({self.g_wd}, {self.g_ht}); '
f'(x_div, y_div)=({self.x_div}, {self.x_div}); '
f'do_lbls={self.do_lbls}; '
f"grid={type(self.grid)} {self.grid.shape}"
)
def get_grid_cell(self, ls_x, ls_y, l_lbl=None):
gx = math.floor(abs(ls_x - self.x_s) / self.x_div)
gy = math.floor(abs(ls_y - self.y_e) / self.y_div)
if not self.grid[gx, gy]:
if l_lbl is not None:
self.grid[gy, gx] = l_lbl
else:
self.grid[gy, gx] = 1
return gy, gx
else:
return None, None
Test 1: grid not using labels.
... ...
g_ls = ae_utils.ImageGrid((-15, 15), (-60, -30), g_wd=15, g_ht=15)
# tests
if True:
print(repr(g_ls))
d_tst = [(-14, -31), (-14.5, -31), (14, -59), (14.5, -59.5), (0, -25),
(2.4, -31), (2.5, -59.5)
]
for x, y in d_tst:
rw, cl = g_ls.get_grid_cell(x, y)
print(f"g_ls.get_grid_cell({x}, {y}) = ({rw}, {cl})")
if g_ls.do_lbls is None:
imgplot = plt.imshow(g_ls.grid)
plt.show()
else:
print(g_ls.grid)
And the terminal output follows. I won’t bother with the image.
(mclp-3.12) PS F:\learn\mcl_pytorch\proj7> python autoe.py -rn rek_2 -bs 32 -ep 5
{'run_nm': 'rek_2', 'dataset_nm': 'no_nm', 'sv_img_cyc': 150, 'sv_chk_cyc': 50, 'resume': False, 'start_ep': 0, 'epochs': 5, 'batch_sz': 32, 'num_res_blks': 9, 'x_disc': 1, 'x_genr': 1, 'x_eps': 0, 'use_lrs': False, 'lrs_unit': 'batch', 'lrs_eps': 5, 'lrs_init': 0.01, 'lrs_steps': 25, 'lrs_wmup': 0}
image and checkpoint directories created: runs\rek_2_img & runs\rek_2_sv
slct_ls: <class 'numpy.ndarray'> 331 (331,) [ 16 33 78 132 135]
ImageGrid((x_s, x_e)=(-15, 15); (y_s, y_e)=(-60, -30); (g_wd, g_ht)=(15, 15); (x_div, y_div)=(2.0, 2.0); do_lbls=False; grid=<class 'numpy.ndarray'> (15, 15)
g_ls.get_grid_cell(-14, -31) = (0, 0)
g_ls.get_grid_cell(-14.5, -31) = (None, None)
g_ls.get_grid_cell(14, -59) = (14, 14)
g_ls.get_grid_cell(14.5, -59.5) = (None, None)
g_ls.get_grid_cell(0, -25) = (2, 7)
g_ls.get_grid_cell(2.4, -31) = (0, 8)
g_ls.get_grid_cell(2.5, -59.5) = (14, 8)
Test 2: grid using labels.
if True:
# test grid class using labels
g_ls = ae_utils.ImageGrid((-15, 15), (-60, -30), g_wd=15, g_ht=15, lbls=True)
rnd_ndx = cfg.rng.choice(slct_ls, size=10)
for l_ndx in rnd_ndx:
x, y = ls_data['tsne'][l_ndx]
rw, cl = g_ls.get_grid_cell(x, y, l_lbl=ls_data["labels"][l_ndx].item())
print(f"g_ls.get_grid_cell({x}, {y}) = ({rw}, {cl}) => {ls_data["labels"][l_ndx].item()}")
if not g_ls.do_lbls:
imgplot = plt.imshow(g_ls.grid)
plt.show()
else:
print(g_ls.grid.astype(str))
And the terminal output follows.
(mclp-3.12) PS F:\learn\mcl_pytorch\proj7> python autoe.py -rn rek_2 -bs 32 -ep 5
{'run_nm': 'rek_2', 'dataset_nm': 'no_nm', 'sv_img_cyc': 150, 'sv_chk_cyc': 50, 'resume': False, 'start_ep': 0, 'epochs': 5, 'batch_sz': 32, 'num_res_blks': 9, 'x_disc': 1, 'x_genr': 1, 'x_eps': 0, 'use_lrs': False, 'lrs_unit': 'batch', 'lrs_eps': 5, 'lrs_init': 0.01, 'lrs_steps': 25, 'lrs_wmup': 0}
image and checkpoint directories created: runs\rek_2_img & runs\rek_2_sv
slct_ls: <class 'numpy.ndarray'> 331 (331,) [ 16 33 78 132 135]
g_ls.get_grid_cell(-1.2251273393630981, -53.005577087402344) = (11, 6) => 2
g_ls.get_grid_cell(-2.52998948097229, -52.90231704711914) = (11, 6) => 2
g_ls.get_grid_cell(4.157108783721924, -34.331321716308594) = (2, 9) => 8
g_ls.get_grid_cell(11.130467414855957, -30.141386032104492) = (0, 13) => 8
g_ls.get_grid_cell(1.6500715017318726, -30.146703720092773) = (0, 8) => 8
g_ls.get_grid_cell(7.7101240158081055, -30.303123474121094) = (0, 11) => 8
g_ls.get_grid_cell(-10.707687377929688, -47.055824279785156) = (8, 2) => 2
g_ls.get_grid_cell(-11.207798957824707, -47.016632080078125) = (8, 1) => 2
g_ls.get_grid_cell(-11.60817813873291, -57.94465637207031) = (13, 1) => 2
g_ls.get_grid_cell(-10.707687377929688, -47.055824279785156) = (8, 2) => 2
[['' '' '' '' '' '' '' '' '8' '' '' '8' '' '8' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '8' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '2' '2' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '2' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '2' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']]
When using the training set to generate the visualization the above becomes.
(mclp-3.12) PS F:\learn\mcl_pytorch\proj7> python autoe.py -rn rek_2 -bs 32 -ep 5
{'run_nm': 'rek_2', 'dataset_nm': 'no_nm', 'sv_img_cyc': 150, 'sv_chk_cyc': 50, 'resume': False, 'start_ep': 0, 'epochs': 5, 'batch_sz': 32, 'num_res_blks': 9, 'x_disc': 1, 'x_genr': 1, 'x_eps': 0, 'use_lrs': False, 'lrs_unit': 'batch', 'lrs_eps': 5, 'lrs_init': 0.01, 'lrs_steps': 25, 'lrs_wmup': 0}
image and checkpoint directories created: runs\rek_2_img & runs\rek_2_sv
slct_ls: <class 'numpy.ndarray'> 1651 (1651,) [ 23 32 37 63 159]
g_ls.get_grid_cell(-0.9511213302612305, -53.56804275512695) = (11, 7) => 7
g_ls.get_grid_cell(12.136856079101562, -32.515438079833984) = (1, 13) => 9
g_ls.get_grid_cell(-6.958099365234375, -34.31993103027344) = (2, 4) => 8
g_ls.get_grid_cell(9.895232200622559, -56.564056396484375) = (13, 12) => 5
g_ls.get_grid_cell(14.372328758239746, -43.96846389770508) = (6, 14) => 5
g_ls.get_grid_cell(-6.453700542449951, -55.774295806884766) = (12, 4) => 2
g_ls.get_grid_cell(7.026605606079102, -55.31654739379883) = (12, 11) => 5
g_ls.get_grid_cell(11.089277267456055, -38.57889175415039) = (4, 13) => 5
g_ls.get_grid_cell(10.472847938537598, -58.84883117675781) = (14, 12) => 5
g_ls.get_grid_cell(-7.990347862243652, -52.02255630493164) = (11, 3) => 2
[['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '9' '']
['' '' '' '' '8' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '5' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '5']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '' '' '']
['' '' '' '2' '' '' '' '7' '' '' '' '' '' '' '']
['' '' '' '' '2' '' '' '' '' '' '' '5' '' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '5' '' '']
['' '' '' '' '' '' '' '' '' '' '' '' '5' '' '']]
And you can already get an idea of what the model will put where for our selected range of points.
Et fini!
Well that’s a day’s work and getting to be a lengthy read. So, I think I will call it a day and close this post.
Will hopefully get the visualization completed next time.
You likely pay more attention to the details than I did when starting this mini project. I also hope, in the end, you find success if perhaps you don’t always keep the details in mind.
Resources
- matplotlib Image tutorial
- numpy random.Generator.choice
- numpy bytes to plain string