Ok, going to continue with daily and historical temperature and weather conditions. I am thinking that some sort of chart showing the historical weather condition for the current day might be of some interest in the dashboard. What I am thinking of is showing a bar chart indicating how often any particular condition was recorded.
Weather Condition Counts
So, let’s see if we can get those counts for the current date across the years in the temperature table. Turns out to be rather straightforward use of COUNT and GROUP BY.
if tst3:
c_md = time.strftime("%m.%d", time.localtime())
print(f"current month/day: {c_md}")
q_wcnt = f"""SELECT condition, COUNT(*) as cnt
FROM {t_nm}
WHERE SUBSTR(datetime, 6, 5)='{c_md}' AND condition!=''
GROUP BY condition;"""
w_cnt = rfall.qry_pd(q_wcnt)
print("\n", w_cnt)
And␎
(dbd-3.13) PS R:\learn\dashboard\utils> python weather_db.py
current month/day: 09.24
condition cnt
0 mainly clear 1
1 mostly cloudy 2
2 partly cloudy 1
Trouble
Well, not many entries with the weather condition for that date. Wonder why? Well, here’s a couple examples of why.
<h3>2021.09.24</h3>
<ul>
<li>Mostly cloudy. No rain recorded last 24 hours.</li>
<li>07:24 & g=11°C / 83%.</li>
... ...
<h3>2020.09.24</h3>
<ul>
<li>Raining hard. 49.5mm recorded for yesterday, 23.7mm so far today. Total for yesterday and today currently at 23.7mm, with plenty more to come next few to several days.</li>
<li>Around 08:15, it started raining very hard. In the time it took for me to go down and get a cup of coffee the rate of rainfall went from 2.1 mm/hr to 7.9 mm/hr. It eventually reached a high of 8.1 mm/hr about 10 minutes later. So, if my arithmetic is correct it was effectively raining about 38 mm/hr for that 10 minutes or so. Couple of the gutters were definitely overflowing. After that 10 or so minutes, it has virtually stopped.</li>
<li>08:35 & 13.7°C. 24hr low 13.6°C, high 16.8°C, reset.</li>
Now I sure don’t want to go back, refactor the parsing code, and rebuild the temperature table from scratch. But, I feel I really should do something about the avaiable data not being in the table.
Get Missing Weather Condition Data
I am going to look at going through the files looking for any rows that contain something that looks like a weather condition for any given day. New generator? While developing the code I will print the date, note and identified weather condition to the terminal. And, see what is a valid condition for that date and what really isn’t. Then figure out how to make that distinction in code.
I will start looking at the notes for the year 2021. Once I have some code working, I will see how it handles the year 2015.
Generator
Okay, a new generator it is.
def get_cond_data(rg_fl):
'''Generator for daily note php files, only want certain lines so decided to use generator.
Want any rows mentioning a weather condition. E.G. cloudy, raining, clear
Parameters:
rg_fl: full path to file
'''
w_conds = ["mostly cloudy", "overcast", "partly cloudy", "cloud", "cloudy", "clouds",
"snow ", "snow,", "snowfall", "snowing", "fog", "foggy",
"light rain", "heavy rain", "rain ", "raining", "light snow", "heavy snow",
"mainly clear", "mainly sunny",
"mostly clear", "mostly sunny", "clear ", "sun ", "sunny"
]
with open(rg_fl, "r", encoding='utf8') as ds:
c_dt = ""
for ln in ds:
ln = ln.strip().casefold()
if ln[:4] == "<h3>":
c_dt = ln[4:14]
if any(wc in ln for wc in w_conds):
yield f"{c_dt}: {ln}"
For the test, I have refactored get_weather_cond slightly.
def get_wthr_cond(uc_nt):
"""Check for any weather condition(s) in the passed in daily note
Return the first one found. Empty string if none found.
"""
nt = uc_nt.casefold()
d_cnd = ""
if "mostly cloudy" in nt or "overcast" in nt:
d_cnd = "mostly cloudy"
elif "partly cloudy" in nt:
d_cnd = "partly cloudy"
elif "cloud " in nt or "cloudy" in nt or "clouds" in nt:
d_cnd = "cloudy"
elif "light snow" in nt:
d_cnd = "light snow"
elif "heavy snow" in nt:
d_cnd = "heavy snow"
elif "no snow" not in nt and ("snow " in nt or "snow," in nt or "snowing" in nt or "icy rain" in nt):
d_cnd = "snow"
elif "fog" in nt or "foggy" in nt:
d_cnd = "fog"
elif "light rain" in nt:
d_cnd = "light rain"
elif "heavy rain" in nt:
d_cnd = "heavy rain"
elif "no rain" not in nt and ("rain " in nt or "raining" in nt):
d_cnd = "rain"
elif "mainly clear" in nt or "mainly sunny" in nt:
d_cnd = "mainly clear"
elif "mostly clear" in nt or "mostly sunny" in nt:
d_cnd = "mostly clear"
elif "clear " in nt or "sun " in nt or "sunny" in nt:
d_cnd = "clear"
return d_cnd
And a quick test of the generator and refactored function.
if tst_wcond:
src_ndx = 0
s_pth = tmpr_srcs[src_ndx].__str__()
print(f"{src_ndx} -> {s_pth}")
t_gnr = wd.get_cond_data(tmpr_srcs[src_ndx])
for i in range(8):
w_nt = next(t_gnr)
print(w_nt, "\n\t", get_wthr_cond(w_nt))
And in the terminal, I got the following.
(dbd-3.13) PS R:\learn\dashboard> python data2db.py
0 -> F:\BaRKqgs\gdn\bark_gdn_2015.php
2015.01.01: <li>happy new year! at 07:30 it's -3.6°c, yesterday's high was 1.3°c, overnight low -3.6°c. seems to always get a little cooler as the sun comes up this time of year. both hummingbird feeders are already out. also put some millet on the ground. took forever for them to clean it up yesterday.</li>
clear
2015.01.02: <li>as of around 08:15 light snowfall or icy rain began. currently 0.6°c, 24 hour low -3.6°c, high 2.4°c.</li>
light snow
2015.01.02: <li>light snow continued for a few hours. started turning to slush in mid afternoon.</li>
light snow
2015.01.03: <li>a little dark but under street lights doesn't look like we got more snow overnight. good thing, cause have to take new camry in for snow tires this morning. as of 07:00 1.3°c; 24 hour high 1.3°c, low: 0.4°c.</li>
snow
2015.01.04: <li>started raining last night. continued into the morning, don't expect it to stop anytime soon. but, no snow so far.</li>
snow
2015.01.04: <li>barbara says wet snow started coming down around 09:20. as of 09:38, is 0.9°c.</li>
snow
2015.01.04: <li>near as we could tell from the back door was about 20mm in the rain gauge when it started snowing.</li>
snow
2015.01.04: <li>snow/ice didn't last long, started raining heavily shortly after. continued all day.</li>
rain
And for another year, the following was printed to the terminal.
2 -> F:\BaRKqgs\gdn\bark_gdn_2017.php
2017.01.01: <li>a lot of cloud to start the day. though looks like might be some clear patches above. can't tell if it snowed any more last night. but there's about 10cm of snow sitting on the deck railing out side the kitchen door. barbara headed out before 7 to go grocery shopping. unfortunately, they're not upening until 8am today. nothing posted in flyer or local papers or at the store. says will go again. apparently our street has been plowed, somewhat/partially at least. is in better shape than 156..</li>
cloudy
2017.01.01: <li>going to record all snow on ground as having fallen yesterday (which it most likely did). 08:30, measured 11.4mm center of driveway.</li>
snow
2017.01.02: <li>clear skies overnight (could see a start or two when i looked out). and, with available light, looks to be clear skies to start the day. was about -2.2 when i looked a few minutes ago. not as cool, i think, as was forecast. but, supposed to me cooler next 3 or so nights. possible flurries thursday night.</li>
clear
2017.01.03: <li>clear overnight. clear and cool to start the day. ~-4.2°c around 07:20. just put out the 2nd hummingbird feeder (on garage), as well as put down seed for ground eaters and top up bird bath. barbara put first hummingbird feeder (by kitchen window, only one where she can reach hook) out around 07:00.</li>
clear
2017.01.04: <li>clear and rather crisp to start the day (less than -5°c). hummingbird feeders and fresh seed out by 07:05. will need to monitor hummingbird feeders for freezing today. doesn't look like i will be mowing the lawn anytime soon.</li>
clear
2017.01.04: <li>warmed up a touch; but, wow, the roads around here are virtually pure ice (2-3cm thick). makes driving rather worrisome the whole time you're out on the roads. felt myself slipping a number of times yesterday and today as i tried to stop. have gotten a spare for friday doubles at evergreen as forecast has possible flurries thursday night and friday. don't want to be driving across the lower mainland if it's snowing.</li>
snow
2017.01.05: <li>went out today and got a couple bags of hulled millet (we ran out a couple days ago), and grabbed some veggies at the same time — supposed to snow tonight and tomorrow. brought subs home for lunch. and, just finished a pot of veggie soup, to which i added the left over sausage bake, chili and spaghetti and meatballs. should be interesting? driving conditions on anything but major streets is hazardous at best. almost all have lots of ice, with perhaps a touch of snow or soft ice on top. got a spare for tomorrow so that i wouldn't have to make the trip across the gvrd to get to evergreen (north van). hopefully we are good until next week and can stay home for the next few days.</li>
snow
2017.01.06: <li>very light dusting of snow visible on the driveway early this morning. cloudy, but currently not snowing or raining. 3-4° warmer than yesterday morning.</li>
cloudy
Pretty clearly my lack of consistent note taking is definitely making parsing the weather condition out of these files extremely difficult.
Rather than give up, I am going to refactor the test code to run a whole year and write the output to a new file rather than the terminal. I will then manually go through the file, determine the correct condition, and delete the note. I will assign the condition, if found, to the specified date. If not found, I will delete date and all. In the end I should have a CSV file with the date and associated weather condition for those dates where I can determine the condition. Will take some time per year, but…
Once done, or year by year, I will use the CSV file to update the database table.
Refactored File Processing Code
Okay, here’s the refactored test code that processes a year, writing the output to file.
if do_yr_wcond:
src_ndx = 0
s_pth = tmpr_srcs[src_ndx].__str__()
f_yr = tmpr_srcs[src_ndx].__str__()[-8:-4]
fl_pth = cwd/"data"
fl_nm = f"wcond_{f_yr}.csv"
d_pth = fl_pth/fl_nm
print(f"{src_ndx} -> {s_pth} to {d_pth}")
t_gnr = wd.get_cond_data(tmpr_srcs[src_ndx])
f_wc = open(d_pth, "w", encoding='utf8')
for w_nt in t_gnr:
w_conds = get_wthr_conds(w_nt)
if w_conds:
f_wc.writelines([f"{w_nt}\n", f"\t{w_conds}\n"])
f_wc.close()
While messing around with the above, I kept refactoring the get_weater_cond function. Eventually I completely rewrote it.
def get_wthr_conds(uc_nt):
"""Find all potential weather conditions in the passed in daily note
Return array of found items. Empty array otherwise
"""
nt = uc_nt.casefold()
d_cnd = []
if "mostly cloudy" in nt or "overcast" in nt:
d_cnd.append("mostly cloudy")
if "partly cloudy" in nt:
d_cnd.append("partly cloudy")
if "cloud " in nt or "cloudy" in nt or "clouds" in nt:
d_cnd.append("cloudy")
if "light snow" in nt:
d_cnd.append("light snow")
if "heavy snow" in nt:
d_cnd.append("heavy snow")
if "snow " in nt or "snow," in nt or "snowing" in nt or "icy rain" in nt:
s_cnt = nt.count("snow")
if s_cnt == 1 and "no snow" not in nt:
d_cnd.append("snow")
if "fog" in nt or "foggy" in nt:
d_cnd.append("fog")
if "light rain" in nt:
d_cnd.append("light rain")
elif "heavy rain" in nt:
d_cnd.append("heavy rain")
elif "rain " in nt or "raining" in nt:
r_cnt = nt.count("rain")
if r_cnt == 1 and ("no rain" not in nt and "rain gauge" not in nt and "stopped raining" not in nt):
d_cnd.append("rain")
elif "mainly clear" in nt or "mainly sunny" in nt:
d_cnd.append("mainly clear")
elif "mostly clear" in nt or "mostly sunny" in nt or "partly sunny" in nt or "partly clear" in nt:
d_cnd.append("mostly clear")
elif "clear " in nt or "sun " in nt or "sunny" in nt:
d_cnd.append("clear")
return d_cnd
And, I decided to add getting the time if present in the note in the generator output.
with open(rg_fl, "r", encoding='utf8') as ds:
c_dt = ""
for ln in ds:
c_tm = ""
ln = ln.strip().casefold()
if ln[:4] == "<h3>":
c_dt = ln[4:14]
rgx = r"^.*?(\d{2}:\d{2})"
rx = re.compile(rgx, re.IGNORECASE )
mtch = rx.search(ln)
if mtch:
c_tm = f" {mtch.group(1)}"
if any(wc in ln for wc in w_conds):
yield f"{c_dt}{c_tm}: {ln}"
And here’s some of the lines from the file.
... ...
015.01.04 09:20: <li>barbara says wet snow started coming down around 09:20. as of 09:38, is 0.9°c.</li>
['snow']
2015.01.04: <li>near as we could tell from the back door was about 20mm in the rain gauge when it started snowing.</li>
['snow']
2015.01.04: <li>snow/ice didn't last long, started raining heavily shortly after. continued all day.</li>
['rain']
2015.01.06 06:30: <li>stopped raining overnight. chance of drizzle this morning according to forecast. according to weather site, is mostly cloudy, but too dark for me to tell (06:30, been up since 05:30, awake since about 05:00 — miss sarah started early today).</li>
['mostly cloudy', 'cloudy']
2015.01.07 14:17: <li>cloudy, foggy, no precip. a little late w/the temps. 14:17: 6.2°c, low 3.5°c, high 8.8°c.</li>
['cloudy', 'fog']
... ...
2015.12.08 08:00: <li>fairly heavy rain to start the day. as of 08:00 ~84.5mm in gauge. so ~27mm since yesterday morning.</li>
['heavy rain']
2015.12.09: <li>overcast and some wind to start the day. rain seems to have stopped for a bit.</li>`
['mostly cloudy', 'rain']
2015.12.10: <li>light rain to start the day. bit of wind early on as well.</li>
['light rain']
2015.12.11: <li>cloudy start to the day. but as i write this no precipitation.</li>
['cloudy']
2015.12.11 10:01: <li>10:01 & 6.8°c. 24hr low 6.3°c, high 8.2°c, ~6mm in gauge (asftbd), so ~3.5mm last 24 hours or so. actually have sunlight at the moment, with clear skies to the north and some cloud to the south.</li>
['cloudy', 'clear']
And, here’s what the first several lines of the file look like after I manually edited them.
2015.01.01 07:30,clear
2015.01.02 08:15,light snow
2015.01.04,snow
2015.01.06 06:30,mostly cloudy'
2015.01.07 14:17,cloudy
2015.01.08 09:02,fog
2015.01.09 08:06,cloudy
2015.01.10,cloudy
2015.01.11 06:30,rain
2015.01.12 07:22,fog
2015.01.13 07:57,fog
2015.01.14,fog
2015.01.15,cloudy
Finished for Now
I think that’s it for this post. I am going to contiune working on producing those files and updating the table.
Resources
- Test if string contains element from list
- SQLite Aggregate Functions » SQLite COUNT
Nota Bene
I think I should let you know that I may not be publishing a post for a week or three. I am scheduled for heart surgery this Tuesday. If I am not bumped, it will likely be around a week before I get back home. And, I don’t know how comfortable it will be sitting at the desk typing and such. So, until next time, wishing you the best of the season and much happiness and health in the New Year.
As I celebrate Christmas, a very Merry Christmas to those of you where appropriate.