Solving hard problems
After having completed a very hard task at work, I felt I'd made some changes to my usual workflow that I wanted to document as a reminder for myself for how to approach hard problems again. I have a feeling it will apply to all variants of journey listed above.
Firstly, I realised that I really have a hard time understanding things unless I can break down every single iota of information in the process and build them back up in my mind.
The issue I was working on involved a very old script that was hand-crafted (back in my day code was written by hand and it sure was shitty1). It wasn't written by me, but was actually purchased from the vendor we work with to handle customer routing on the phone (aka an IVR - Interactive Voice Response: the thing that asks you to "press one for customer service, or press two for any other request", to which most people just slam their hand on the keypad in order to get anyone to answer the damn phone). It was a bit of a dogs breakfast and I'd always just assumed it worked. Once upon a time it probably did, but those days were long gone.
Therefore, in order to fix and improve it, I sat down to learn it. Over. And Over. And Over. And this thing was just so complicated, so filled with undocumented methods and hooks, all part of a scripting framework that just has nothing documented I can read to understand. Trying to patch on top of old code with 800 different exits didn't work. I kept trying to add or adjust parts that in my head made sense, but ended in tears (not quite literally, but I was getting close).
Fed up with trying to make bad code work, I did what has always worked and been a pleasure for me in my personal life: I broke it all into pieces and start rebuilding it by adding one block at a time, testing it, adjusting, test again, repeat. What resulted was something I understood, was, dare I say it, well written, and most importantly: it worked.
The change here is that while I could've just hit the existing rust bucket with a sledgehammer until it croaked out a cloud of exhaust that you could "press one" with while trying to breathe and maybe someone ends up joining you to chat in the aforementioned smog-cloud, it would've resulted in a far less successful result. Now we have something that works well and we can extend as we need, but that was only possible by taking the hard path here; something I should've done from the start. Sometimes you just need to start over.
Second, I set myself up for success (by rewarding my brain with pretty colours).
I spent a lot of time researching this issue, which in my line of work mostly means sifting through reams of JSON to find patters that I can manipulate to get the output I want to monitor said system, in order to take action on that system to get things working the way they need to. Observing, understanding, taking action. The key route I focused on this time around, and far more than usual, was to set up my observability environment exactly the way I needed to see changes as fast as possible.
Streamlit is a godsend here and I focused on getting visualisations done quickly. I wanted my feedback loop to be two things: as close to real-time as made sense, and unmistakeable in its readability. If a lever I pulled made a change for the good, I wanted to see that immediately and undeniably. Think about how a stock chart looks, with all the Christmas-coloured lines going up and down: I wanted that.
I have a lazy point in my work sometimes where I make a change, monitor briefly from the raw logs, then close the issue and move on with my life. If it looks good from the flow of data that I can see, then we're good. However, this time, by focusing on the observability and shaping it to look such as it did, when the fix was deployed and green lines just started going up almost immediately, I felt overwhelmed with satisfaction. I'd intentionally set my work up to reward my brain as much as possible, and for that reason I found myself wanting to more deeply understand the problem and to craft a fix that would cover as many edge cases as seemed reasonable, in order to make caveman happy with big green line. And feel good it did.
Lastly, I set myself dedicated alone time to get this done. I actually was so annoyed that I got home at like 6pm this Wednesday and while I'd been working all day on this and other issues at work in the office, I knew that I had a few hours alone while Sofia was out, so I posted up at my desk and just kept working on this issue. Uninterrupted by Slack, no people talking around me, the usual office-buzz that I actually love was replaced by my quiet desk at home and the problem in front of me with no distractions.
That session of two-and-a-half hours led to my biggest revelation, which was that the full-rewrite was necessary, and half of the code being written. It was deeply productive and I wish I had done this weeks ago, but I don't tend to mix business and pleasure. I love going to the office but I also love having a complete separation of work and home, so I almost never work at home, and even more rarely do I work late into the night. Bringing work home with me is a rarity. I need to find a way to harmonise these things.
That's it! It probably seems obvious to a lot (or even all) of you, and to some degree I know all of these lessons already, but they were approached a lot more intentionally this time around and it worked so well, I'm excited for the next hard problem to solve.
I've only been working with code for like two years pre-ChatGPT and honestly, thank god. I far prefer the quality of the code I get now.↩