Setting up a Data Scientist’s Workstation
5 Essential tools to improve productivity by 10x.
As Data Scientists and Machine Learning Engineers, we know that building scalable AI features is no walk in the park. It’s like putting together a puzzle with a thousand pieces, each one crucial to unlocking insights and driving decisions.
From wrangling data to fine-tuning models, the journey is packed with twists and turns. With the right tools in our arsenal, we can streamline this process and turbocharger our productivity. 🚀
In this blog, I’ll be sharing my favorite tools that have been an absolute game changer in my quest for efficiency. So grab your coffee, and let’s dive into the wonderful world of tools that can make your data journeys smoother than ever! 🛠️💻
Oh-my-Zsh:
Zsh is a shell, just like bash or fish, which interprets commands and runs them. Oh My Zsh is a framework built on top of zsh that is structured to allow it to have plugins and themes. You can use zsh without Oh My Zsh, but you can’t use Oh My Zsh if you don’t have zsh.
Oh-my-zsh can be installed using curl
sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"It can also be installed using wget as mentioned on the official website.
Plugins:
OMZ supports a ton of exciting plugins to improve developer productivity. The plugins are listed by categories making it easy to navigate and explore. Here are a couple of my absolute favorites:
web-search: This plugin allows you to seamlessly perform web searches on your favorite Search Engine, right from your terminal. This includes searching on Google, Bing, DuckDuckGo, etc. It even allows you to search GitHub and StackOverflow.
autojump: Autojump is a tool that facilitates faster file system navigation. It works by maintaining a database of the directories you use the most from the command line.
git: This plugin provides aliases for GitHub commands. I extensively use GitHub and this plugin covers 90% of the commands that I use regularly.
Themes:
OMZ supports a variety of themes to spice your terminal experience.To enable a theme, set ZSH_THEME to the name of the theme in your ~/.zshrc, before sourcing Oh My Zsh; for example: ZSH_THEME=robbyrussell.
I like simple and elegant designs that don't interfere with my work on the terminal. I am currently using Eastwood.
Troubleshooting:
OMZ maintains a list most commonly reported issues on its FAQ page. This is a great starting point for debugging any issues.
Pyenv:
As a Data Scientist or machine learning Engineer, Python is likely your go-to tool. Ideally, we would like our entire codebase to use the same Python version. But, let’s face it: real-world scenarios often demand juggling different Python versions.
The challenge? Managing and switching between these versions can become a major bottleneck, causing headaches and throwing unexpected errors into the mix. These hiccups not only dent productivity but also put a damper on progress.
Pyenv is a simple Python version management tool that lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
What Pyenv does?
- Lets you change the global Python version on a per-user basis.
- Provides support for per-project Python versions.
- Allows you to override the Python version with an environment variable.
- Searches for commands from multiple versions of Python at a time.
Learn more about how it works here
Installation:
brew update
brew install pyenv
# Python dependencies
brew install openssl readline sqlite3 xz zlib tcl-tkSetup your shell environment for Pyenv
This will ensure that the Python version is automatically updated based on global/local choice in a new terminal session.
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
source ~/.zshrcInstalling a specific Python version using Pyenv:
pyenv install <version>
# for instance, `pyenv install 3.11`Setting a Python version as a global choice:
Note that you would have to start a new terminal session for the changes to take effect.
# Setting 3.12 as the global choice
pyenv global 3.12
# setting 3.11 as the local choice
pyenv local 3.11
Poetry:
Using the system Python to install and manage Python dependencies can turn into a nightmare. Each DS/ML project may depend on specific Python package versions. Reinstalling the Python packages on demand is neither efficient nor scalable. Virtual environments partially solve this problem by spinning up isolated Python environments. However, users still need to manually create, activate, and manage dependencies. Additionally, Virtual environments are not shareable and reproducible.
Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on using a .toml file and it will manage (install/update) them for you. Poetry converts the list of Python libraries into a lockfile that tracks the Python versions and their dependencies. This ensures that the virtual environment is reproducible on any machine.
Installation:
# ensure that pip is installed
python -m ensurepip
pip install pip -U
# install poetry using pip
pip install poetry If you have never worked with poetry before, checkout their official documentation to get started.
Jupytext:
Jupyter notebooks are bread and butter for Data Scientists and ML Engineers but it is a nightmare to version control or review them. With Jupytext, you can author .py files and simply open them as notebooks. Authoring .py makes it easier to edit and refactor the code in an IDE.
To leverage Jupytext, all you have to do is include it as dependency in the projects pyproject.toml poetry configuration. For instance,
[tool.poetry]
name = "jupyter-playground"
version = "0.1.0"
description = ""
authors = ["Pritish J"]
[tool.poetry.dependencies]
python = "3.12.3"
jupytext = "*"
jupyterlab = "*"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"Tmux:
TMUX is a program that runs in a terminal and allows multiple other terminal programs to be run inside it. Each program inside TMUX gets its own terminal managed by TMUX, which can be accessed from the single terminal where TMUX is running — this is called multiplexing and TMUX is a terminal multiplexer.
Here is a quick before and after comparison:
Follow the detailed installation guide here and get started.
In my experience, diving into TMUX can be challenging at first. However, this cheat sheet provides a solid foundation with the essential commands, making it a great starting point for exploration.
Moreover, I have fine tuned my TMUX configurations to better align with my workflow and preferences. If you are looking to enhance your TMUX setup for a smoother and more visually appealing experience, I would highly recommend the following blog. It offers valuable insights and tips for tweaking TMUX configurations to better suit your needs.
Final Thoughts:
Tools are fantastic, but wield them wisely. Its tempting to install every shiny tool out there, but remember, unnecessary installations can clutter your workspace and even slow you down. Instead, opt for a lean, purposeful toolkit that aligns perfectly with your workflow and objectives. Each tool should serve a distinct purpose, enhancing your productivity and efficiency without overwhelming you with unnecessary features.
Finally, if you have a secret weapon in your toolkit that wasn’t covered in this article, share it with the community via comments. Let’s keep the conversation going and help each other uncover these hidden gems that make our journeys enriching.
