Will there ever be a week again where all setting up for this show is completely, 100% uneventful? I really hope there will be because I’m not sure how much more of this stress I can handle…
But I really should start from the beginning. I have small collection of shell and Python scripts that collect HTML, extract JSON from it, and convert that to CSV. This collection is situated on my server, and runs several times a day.
On my workstation at home I have a project with the chart generator tool and it needs the JSON data generated by the script collection. For easier handling, I rsync all the data files from the server to my workstation when I want to generate the charts for a new week.
So, on monday I rsync’ed all the files, ran the tool, and was met with an exception. A short investigation yielded the result that the JSON file had a length of 0 bytes. Well, that’s no good!
I took a look at the accompanying HTML file and it was full of data… and it looked different, like, structurally. It finally happened, Beatport changed their website! (On a sidenote: the website now is actually usable, before you had to wait forever for pages to load but now it’s way faster than that!)
The script that extracted the JSON lived on the server so I hopped over there and changed the script to match the new format of the HTML. It’s way easier to work with now! Easy change.
I had to change a whole bunch of files, though, so I whipped up a bit of shell:
for h in *.html; do
j="${h/html/json}"
./extract-json-from-html.py < "$j" > "$h"
done
This script will extract JSON data from an empty JSON file and write the results (which will be empty) to the HTML file.
Sounds wrong? That’s because it is wrong. It is even maximally wrong, the kind of wrong where you not only waste time and other resources but also lose the possibility to fix your mistakes because you mangled/deleted your source files.
I didn’t notice, though. I fired up the rsync script on my workstation, restarted my chart generator and was expecting different results — but… nope. The JSON was still empty. What?
Oh… oh, I see. Fuck.
FUCK.
Not only did I manage to delete the HTML files on the server, no, I went right ahead and removed them from my workstation as well! 🤦
I was angry and frustrated after that and had to step away from the computer so instead of shredding everything that uses electricity I went to have some dinner. During dinner not only did I feed my body, but apparently also my mind. “Hmm, didn’t I add a new disk to my NAS a couple of weeks ago? Yes, I believe I did!”
How is that relevant? Well, you see, the NAS is where Time Machine makes its backups to. And before I added the new disk the space on the NAS was full and I couldn’t make any backups but since I added the new disk I can finally make backups again! (You really, really should have backups.)
The directory where I rsync’ed the files to regularly was also being backed up so at this point it was a breeze to go back to earlier in the day and restore all the HTML files, saving the day and the show. I regenerated the JSON files, changed the chart generator to parse its data from the new JSON structure, and the rest, as they say, is history.
Phew, what a story! Anyway, Favourite of the Week? Easy! Time Machine! ❤️ Oh, I have to pick a track? Oh well, Nhato it is, then… 😁
The rest of the Nhato EP is also still in there, and it’s all in one single, easy-to-listen-to block! 😄