Year 2021 in review - a year of transitions

[ reflection  ]

A reflection post on year 2021, and how COVID-19 affected my speaking, career and life in general


One word to describe 2021: Transitions.

At the start of 2021, I moved to a new role in the fintech industry and started learning Scala for data engineering.

Offices were starting to reopen at that time, and the pandemic situation seemed to be improving.

I set myself a goal to get certified in Kubernetes. A few months later, I boldly told my boss that I would clear my AWS Certified Solutions Architect (Associate) certification within the year to demonstrate my cloud expertise.

To dedicate more time towards my personal learning and career goals while maintaining my relevance in the tech and data community, I decided to focus on quality and limit myself to speaking at 4 tech conferences - no more, no less.

While conceptualising talk proposals for tech conferences, I started work on my functional programming series - intensely researching and crafting iterations of each article in the series.

Even with the uncertainties resulting from the highly-dynamic pandemic situation, I kept pushing myself relentlessly to pursue my goals in a bid to accelerate my career progression.

I achieved some of my 2021 goals - but at what cost?


Recap of Year 2021: Speaking (and Writing?)

At the start of 2021, I decided to focus on quality and aim to deliver talks in 4 tech conferences.

In year 2021, I have given a total of 7 talks, including sharing sessions and a panel discussion:

Transitioning Behind the Scenes

Where: FOSDEM 2021 Python Devroom

When: 21 February 2021

Talk: Is Rainfall Getting Heavier? Building a Weather Forecasting Pipeline with Singapore Weather Station Data

I submitted two proposals for FOSDEM 2021 Python Devroom based on my talks for PyJamas 2020 - the weather forecasting talk was the one that got accepted.

I had fun pre-recording this talk in early January 2021, especially since I had no intention of giving the same talk again. It was meant to be a portfolio talk to showcase my interests in environmental issues in combination with my technical skills in data manipulation and time series analysis/forecasting, at a particular stage in my career where I’m seeking to differentiate myself as a reasonably competent data professional.

This was also what made me decide to discard my hierarchial indexing talk idea for good (I concluded that it was too weak for conferences) and start desperately brainstorming for new talk ideas for year 2021.

As this is a pre-recorded talk with live Q&A, no comments about live audience response or speaking performance. The few people in the live Q&A did give pretty kind comments though.

Exploring new paradigms (while trying to regain my spark)

Talk: Designing Functional Data Pipelines for Reproducibility and Maintainability

If “Speed Up Your Data Processing” represents my ambition as an up-and-coming tech speaker and data professional, “Designing Functional Data Pipelines for Reproducibility and Maintainability” represents my growing pains in staying relevant and keeping the momentum going.

This talk is designed as a taster for my ongoing series on learning functional programming and Scala, which is what I have been learning since 2021 (and will be focusing my efforts on in 2022). As 2021 also coincided with the release of Python 3.10 which includes new functional programming features such as structural pattern matching, it was also an opportune time for me to introduce structural pattern matching as one of the key functional design patterns that can be used in designing data pipelines in Python.

I took weeks to craft an initial talk outline, picturing myself giving the talk to reinforce my learnings. I spent close to a week working on the visuals in my talk slides using 90% Excalidraw and 10% Google Slides - designed to fit my own mental imagery of these technical concepts so that I can incorporate them into the talk I had been iterating in my head.

In my mind, the success (or failure) of this talk would be an important barometer of my continued relevance - if this talk resonates well with the audience, publishing more follow-up articles for my ongoing functional programming series is likely to contribute high value to the tech community. Otherwise, I would take it as a sign to scale back and pivot towards other pursuits that contribute better Return On Investment (ROI) for my career and personal life.

After delivering this taster talk on functional programming at 3 major Python / data conferences with some modest success, I believe that the Python and data community will become more receptive towards practical application of functional programming in designing production-ready data pipelines. With the emerging interest in data lifecycle topics such as data management and reproducibility of data pipelines, I look forward to driving the conversation in the data community through various mediums - one step at a time.

Why I have decided to cut down on speaking (for now)

In my 2020 speaking review, I mentioned how the temptation of giving as many talks as possible with virtual conferences becoming the “new normal” led to some form of burnout as I tried to balance between conference speaking, my career, and my personal life.

The feelings of burnout and fatigue did not get any better. In fact, my burnout got even worse in 2021.

I wondered if I was putting too much pressure on myself by focusing on quality over quantity for my talk proposals this year, especially after getting Standby-ed for PyCon US 2021 and rejected for PyCon AU 2021 early in the year. The Standby outcome for PyCon US 2021 was a heavy blow for me, as it marked the second time I fell short of secure a speaking slot in a US-based tech conference (I had previously been waitlisted for OSCON 2020 - the one that never happened).

I wondered what’s the point in me speaking to the webcam for a virtual conference when it’s highly likely that the engagement level will be depressingly low, ROI would be insignificant relative to the effort put into speaking, and that no one except myself would watch the talk replays.

When I did get accepted for both EuroPython and PyData Global for the second consecutive year, I still experienced immense anxiety leading up to the live sessions, wondering if yet another technical issue would threaten to destroy whatever reputation I had (which is admittedly not much) as a tech speaker (it did nearly derail my EuroPython talk this year).

Looking at the engagement rate and talk views on YouTube, it seems that I wasn’t reaping as much direct ROI from speaking at online conferences compared with my base activities - being good at my day job, writing quality articles (occasionally) and pursuing certifications that are relevant to my career path as a data professional.

Preparing new talk material is very time-consuming, requires a lot of work (at least 40 hours from talk preparation alone!), and can result in immense anxiety all the way from the talk preparation process to actually delivering the talk - and your talk is unlikely to get much traction even with your talk recording made available on YouTube unless you have already built sufficient credibility within the tech community.

While writing quality tech articles also requires a lot of work, the ROI seems greater and more scalable in the long run even though I have been sporadic in publishing my writing. In terms of direct ROIs for my writing, my blog posts from 2 years ago are still generating traffic to my portfolio website and I am receiving more inbound connections (and opportunities) from my writing compared with my conference speaking.

Writing has also been useful in helping me consolidate my thoughts and learnings at my preferred pace and depth with relatively less anxiety (though I still get plenty of anxiety from the fear of getting bashed for my writing), compared with conference speaking where I would have to focus on “key takeaways” and worry about tech setup while also worrying about how the audience would perceive my talk.

Conference speaking has indirectly helped in building up my credibility as a data professional and opening doors in my career - especially in the first 2 years. However, I’m not getting as much ROI in terms of networking interactions in online conferences, which has been adversely affected due to the global pandemic.

With these considerations in mind, I have decided to cut down on conference speaking in 2022 and focus more on creating quality content from my writing to increase direct ROI for my career and personal life - while using conference speaking as a force-multiplier to enhance the ROI from my base activities.

Recap of Year 2021: Career and Personal Life

Due to the global pandemic and after deep reflection on my career alignment, I took the leap and joined a fintech scale-up as a mid-level Data Engineer at the start of 2021.

In 2020, I suffered a pay freeze and a subsequent pay cut due to the adverse impact of the global pandemic on the aviation industry. I had already felt growing misalignment in terms of career direction and organizational culture, and desired more room for growth and impact in my career to accelerate my career progression towards seniority. A few weeks after the pay cut was announced and before it took effect, I accepted the offer from the fintech scale-up with the hope that it would provide the rejuvenation I needed in my career trajectory.

At the point of accepting the offer, I decided: Since I have already made the decision to leave the perceived stability of a government-linked corporation (which turned out to be an illusion) for a relatively riskier career option in a different industry, I need to have a plan to maximize my chances of success in the riskier option so that I can reap better returns on my career and financial outcomes.

Fueled by a fresh start, I sought to get up to speed faster to secure my footing in my new role - especially during my first 90 days. Before my first day in the new role, I reached out to two people for fresh perspectives on navigating my new role: a seasoned data leader with an entrepreneurial streak, and a former medtech startup CTO on sabbatical.

I took a one-week break between roles to catch up on my reading, drink plenty of tea while reflecting on my career journey, and prepare myself for my new role. I could have taken another week to enhance my preparation, but I preferred to get through my probation period as soon as possible in case the role did not work out for me (thankfully, I managed to survive my first 90 days with flying colours).

Starting from day one, I took notes of my learnings and work progress as much as I can while getting used to a different tech stack in a vastly different work environment - especially during my first 90 days when I needed to demonstrate my value to the company as quickly as possible.

The learning curve was really, really steep - I learnt Infrastructure-as-Code using Terraform to manage the AWS infrastructure of the data warehouse, and learnt (and still learning) how to write functional Scala code the hard way while working on the existing data pipelines. With my intuitive grasp of complex technical concepts and plenty of hard work, I achieved two certifications alongside my work commitments in 2021: Certified Kubernetes Application Developer (CKAD) and AWS Certified Solutions Architect - Associate.

The Perils of Hitting the Seniority Cliff

As I continued honing my technical expertise in data engineering and demonstrating initiative in my role, more inbound opportunities came knocking - including direct outreach from talent sourcing in higher-tier companies (which I was initially skeptical about, but turned out to be legitimate opportunities). With greater confidence in my technical capabilities and value creation to the company (plus other internal factors at play), I initiated a quarterly review with my manager with the intention of discussing my career path for promotion to the next level and proposing a pay raise.

The issue of leveling and career progression came to a head when I received an unexpected offer for a Senior Data Engineer position from a higher-tier (and well-known) tech company just before my scheduled quarterly review in end September.

Within less than a year, I had to make another difficult game-changing career decision - to achieve seniority as soon as possible by changing companies and tech stack with better overall compensation package, or to play the longer game by sticking to my original plan to seek a promotion in my current company and deepening my expertise in the existing tech stack so that I can get a better offer down the road.

With this decision as the backdrop, I began my first-ever career negotiation. And oh my, it was a half-botched first attempt - while my manager acknowledged the additional value I bring to the company, he felt I wasn’t quite ready for a promotion yet! :disappointed:

After a series of hard-fought negotiations and lengthy discussions with my “personal board of directors”, I had to face the difficult reality within myself that it was not the right time for me to change companies yet from a longer-term career perspective - even though the offer was an up-level from a higher-tier company I had aspired to be a part of even before I entered the data industry. While it was painful to make the difficult decision to decline the offer due to timing despite the better overall compensation package, I do hold hopes of eventually joining the team I interviewed with when the timing is better aligned with my career plans.

Even though I chose to stay in my current role at that point in time, the up-level outside offer and subsequent conversations about my career path provided unequivocably clear signals that I am nearing a seniority cliff at mid-level - and I do have to achieve seniority as soon as possible to remain relevant in my career (or risk being made obsolete by younger mid-level engineers). The inconvenient truth about my career situation (which I’m still learning to reconcile with) is that I may have crossed the seniority bar in terms of technical expertise, initiative and external visibility - but that alone is not going to be enough to overcome the steep seniority cliff and land in a comfortable position at the next level if I do not get the sponsorship from key decision-makers to formally recognize my seniority in terms of leveling.

The months after that were particularly rough for me at all fronts. The pressures of trying to overcome the seniority cliff to gain credibility while being fully remote became so intense that my physical and mental health started showing warning signs.

With the mounting internal and external (though implicit) pressures to achieve seniority as soon as possible, I started perceiving weekly standups as high-stakes events where my performance would be scrutinized for the slightest imperfection - especially after realising the disconnect between how I perceive my performance and how my superiors perceive my performance.

It did not help that the team dynamics of these weekly meetings changes erratically depending on the skip-level manager’s presence even if their primary intent of attending these meetings wasn’t to judge performance. Whenever the skip-level manager attends the weekly standups (which is not every week), it subconsciously becomes a status update meeting where the candidness of sharing progress blockers drops substantially and the focus shifts towards justifiying individual performance. Not only did the shift in team dynamics create process inefficiencies resulting from progress blockers not being surfaced in time (an unintended impact from the skip-level manager’s presence), perceiving the subtle shift in team dynamics also created additional pressure on myself to “prepare for weekly performance reviews” to justify my performance during each standup meeting.

I became increasingly frustrated with progress blockers not being surfaced in time, as it was starting to affect the completion of my yearly performance objectives. After a particularly tense standup meeting caused by delayed communication of blockers, I became even more anxious about these meetings and worried obsessively about how my performance and credibility were being perceived by my superiors.

(For those who are wondering if I am voicing my unhappiness with my current team, this is not the case - every Agile team has their own standup anti-patterns that require tuning. I have been very candid with my colleagues about the standup situation in my team and its impact on my mental health, and have consulted tech/data colleagues within and outside the company about their team dynamics before arriving upon these conclusions.)

To deal with the worsening stress and anxiety about my career progression, I started introducing longer-distance runs of at least 5km into my weekly exercise routines in order to clear my mind. While having more frequent runs did help slightly in improving my mood for the day, the feelings of anxiety kept coming back the next day in anticipation of the “weekly status update meeting” and worrying about whether I have made enough progress to show for the week.

I started experiencing intermittent heart stops by mid October - triggering a more comprehensive health screening that detected several minor abnormalities despite maintaining a weekly exercise routine. While the abnormalities are detected at a very early stage and the prognosis for recovery is great (which is good news), these abnormal findings require further specialist follow-ups and affect my insurability.

By November, I was effectively balancing between preparing for AWS Certified Solutions Architect Associate exam, dealing with the uncertainties resulting from a health scare, handling speaker prep for PyCon APAC and trying very hard to put up a strong front at work for the sake of my career progression.

After passing the AWS Certified Solutions Architect Associate exam in early December, I started feeling unwell and developed a scratchy throat that escalated into a full-blown cold overnight, ending up with mandatory 5 days of MC that adversely affected my intended work timelines. To minimize further impact on my work timelines, I shifted my Moderna COVID-19 booster appointment one week earlier so that the worst of the booster side effects would coincide with the last 2 days of my 5-day MC (‘cos I’m already unwell but not from COVID-19) - thankfully, the plan worked.

While my regular reviews had been mostly positive, my last two monthly reviews have dealt very heavy blows to my confidence as a data professional - calling them “failures” would be an understatement when the consequences could potentially be a deadly blow for my career especially at the cliff (made worse by prevalent ageism in tech - which is a topic for another day).

Epilogue

2022 would be a pivotal year for my tech career trajectory, as attaining seniority is my top-priority goal in 2022 and is of paramount importance to my career progression in tech - especially considering my age (I’m getting older as I write) and gender identity (which should not have any bearing on my commitment towards my career).

If periods of poor overall health and factors outside of our own control are not valid reasons for troughs in performance standards in the corporate world and especially in an industry as Darwinian as tech, then I cannot be faulted for being particularly harsh on the output that I produce with whatever effort I could muster.

Given that I do have to achieve seniority as soon as possible in order to secure my technical career path and survive in the tech/data industry, am I really being too harsh on myself by putting in my all towards accelerating my tech career progresion?

Have I made the right decision to extend my timeline to seniority by staying in my current role?

How can I remain relevant in the tech and data community and optimize how I build credibility without getting severely burnt out at all fronts?

These are the questions from 2021 that I continue to bring with me heading into 2022.

Written on January 2, 2022