The engineering team at Planet Argon continuously works on ways to streamline and improve our processes. One example of this was migrating all of our client applications from Bitbucket to GitHub to have them all centralized. We also wanted to remove any third-party tools we used for CI/CD and utilize GitHub actions instead. What I thought would be a reasonably straightforward process turned into a Mini Linux Server Bootcamp, full of challenges, mistakes, unraveling, building, and valuable lessons.
Learning Linux is something most computer science or experienced developers know, but it's very intimidating to people who come from boot camps (like me). In sharing my personal experience, I hope it inspires you to take on challenges and work through the process with your team.
Initial Apprehensions
When the challenge of migrating to GitHub landed on my desk, it seemed easy when looking through the GitHub documentation. I confess, though, that I underestimated the complexity. My familiarity with the console and basic commands hardly prepared me for the intricate dance in the Linux realm. However, I was open to learning and determined to overcome any obstacle.
Repo Issues
The first issue we ran into pretty quickly was the git history. We received an error when we tried to import the code into GitHub. The repository had some credentials that were accidentally added over ten years ago, and GitHub didn't appreciate that. This one was easy to fix, though. My Senior Engineer found a tool called BFG Repo Cleaner, which removes troublesome 'blobs' from your repository. The troublesome file disappeared with a few simple commands, leaving the rest of our code intact. Here are some of those commands.
# Get a mirrored clone of the Bitbucket repo
git clone --mirror git@bitbucket.org:your_repo.git
# Delete the file across all branches
bfg --delete-files bad_files.bad your_repo.git
# Create the new empty repository in GitHub
# cd into mirror git repo
cd your_repo.git
# Run some git cleanup commands (from docs 🤷🏻♂️)
git reflog expire --expire=now --all && git gc --prune=now --aggressive
# Add our new git remote
git remote add github git@github.com:your_new_repo.git
# Push all the branches (mirrored) to the github remote
git push --mirror github
BUT with this tactic, we're rewriting git history. The file ceased to exist, and so did the commits associated with it. This new history also means we can't change the git origin of our app. None of the commit IDs match anymore. They're technically not the same repo.
But that's easy to fix, right? We can just add the new application to the server and have it switch places with our original. RIGHT?! And here's where things start to get shaky.
Is It All Ruined?!
As you can imagine by this point, as someone who only knew how to run a console on the server or was still impressed that I could duplicate files in the command line like this cp old_file.cpp new_file.cpp (ok, who am I kidding, I still think running this command is cool), I was FREAKING OUT at the thought of having to replace our entire production application.
But we pushed on because that's what we engineers do; we hit roadblocks and learn how to overcome them. 🥲
Cracking the Code
The next issue was some SSH key incompatibilities between GitHub and our server. Github only liked Ed25519 keys, but our older Linux server struggled with those. One of our Senior Engineers, Sergiu Truta, figured it out, but it wasn't straightforward. Now that we had the Ed25519 keys installed and enabled a connection to GitHub, I went to the /tmp folder and tried cloning the repository. Luckily, that went smoothly.
Hours of perseverance, paired with the wisdom of seasoned peers, finally cracked the code. Successfully overcoming these obstacles elevated my confidence in managing server-related tasks.
Deployment
Here is where we saw some issues with the SSH keys. Just for context, this application uses Capistrano for deployment.
While running the deployment process, we ran into many issues. Some (many) of those errors need more helpful descriptions, which meant it was easy to go down the wrong path to discover the root of the problems.
We thought GitHub might not be connecting to the server due to the constant "Access Denied" errors, even though we were able to pull the repository from GitHub on this server with the specific keys 🤷🏽♂️. Many failures all around.
But eventually, what we landed on was that our SSH keys, which we now need both of, needed to be loaded in a particular order. And we also had to turn off our Apple keychain. If not, our keychain injected our RSA key repeatedly without indicating us.
So, we needed to load our RSA to log in to the server and run the deployment through Capistrano. Then, we loaded our ED25519 so the application could connect to GitHub to fetch the appropriate branch and push that into our server.
After a certain point, we also realized that the deployment user lost access to do specific things like destroy old or cached files, but this may be an issue within the original CI/CD pipeline. With all that, it took hours of pairing with more knowledgeable teammates to get to this point. They could connect the dots between these seemingly random strings of events.
Our workflow could have been more streamlined, but it worked. And with continuous collaboration afterward, we were able to iron out some of the kinks. Now, all our devs can deploy more consistently, and we're seeing more helpful descriptions in the logs.
Advice from Senior Devs
During this process, I kept asking my Senior Engineers how to understand DevOps and servers better. They shared with me that improvement comes with experience. One recommendation they all offered was plain and simple: to set up my own server, get on Digital Ocean, get a droplet with minimal configurations, set up a Ruby on Rails project, and deploy it by hand.
I'm almost embarrassed to say it with my ~4 years of experience as a dev, but I honestly didn't even think I could do that. I'm part of the Heroku generation, where we gave this tool our repository, and it deploys it for us. With just one click of a button, I can have a PostgreSQL DB set up for me.
But, do it by hand?! I'm sure they also want me to part the Red Sea, right?!
And as simple of a solution as it is, many of us junior/mid-level engineers shy away from a server because a little voice in the back of our heads says, "Nooo! You'll break this thing, and we'll lose all the PROD data!"
It's ok. Go through the tutorial process.
Again, it's a simple solution that took me to the next level.
Due to a recent conversation with an engineering friend, I realized there was a big difference between the full stack developers of the Bootcamp Era and the developers of the Pre-Bootcamp Era. He said, "I used to call myself full-stack, meaning I could stand up a server and develop an app. Now full stack seems just to mean knowing the front end and back end of the application."
The server felt like something that was always abstract to us in boot camp but was expected of us to understand without fully interacting with it. I'd recommend starting that tutorial and connecting those dots to bridge that knowledge gap. It will give you an eye-opening experience and show you that the application is just a tiny dot in this giant ecosystem.
Lessons Learned
In retrospect, this Linux migration saga was more than a project; it was a transformative experience. From initial trepidation to tackling complexities, each obstacle was a stepping stone toward building my confidence and knowledge.
Do you have a project you need help with? Let's talk about it!